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PREFACE . 

A maj9^' Portion of the Naval Training Equipment Center*s '] 
Avia:tion Wide Angle Visual System (AWAVS) program involves 
behavioral research to' provide a basis for establishing design 
criteria for flight trainers. Because a large number of vari- 
ables will be investigated, . considejrable^ has been 
given to the n\ethodologies appropriate for handling a problem 
of this coinplexity. Dr. Charles W. Simon has/since 1970, been^ 
studying w^ys in which the quality and usefulness of behavioral 
research can be improved through techniques that greatly increase 
the amount of information obtainable from a given amount, of data. 
This contractor report summarizes his views to date concerning 
the application of these "advanced experimental methodologies" 
to the AWAVS program. 

Many of Dr. Simon's technical reports, Listed in the Ref- 
erences/ liave not been widely distributed (although most may be . 
obtained through the Defense Documentation Center or National 
Techinical Information Service) . Therefore, it is hoped that 
this report will be of. benefit not only to those interested in 
the AWAVS program but also-to those who have not yet ^ been exposed 
to^ his Work. Althoiigh not expressly intended as a primer for 
those unfaniiliar with the research paradigm Dr. Simon advocates, 
portions of this report should be helpful to the new reader, in 
particular f Section II discusses the advantages of the multif actor 
approach to research, and Section V provides an illustrative ex- 
ample. A Glossary has also been provided. 

" ■ • ■ , ■ \ "« 

The assistance of Dr. Daniel P. Westra is gratefully acknow- 
ledged for his critical review of T:his report and fdr his helpful 
suggestions. • ^ — ^ ^ 

Stanley C. Collyer 
Scientific Officer 
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SECT TOM I 

INTRODUCTION • , ■ 

• The Naval Tra.ininq Equipnient Center is building a sdphistica 
ted pilot training simulator which focuses on advancing the state 
of-the-art of, the visual .system. '..The aviation wide angle visual 
^system (AWAVS) along with a s i ;v:-dO'jreos -of --f reedom mot ion system 
combine .to provide a highly versatile simulator on wftich complex 
behavioral research «*e?ii be performed. Initially, the primary " 
puroose of such* research will be to exam^ine and optimize the sim- 
.ulator parameters for pilot training in specif ic "carrier-landing 
t^ks. The larg'^ number of parameters that must be irvestigated 
requires tHe use advanced experimental methodologies* for ^ 
studving many factors economically. A discussion of philosophy, 
strategy, and te^hn iques that miuht be employed' on this program 
represents the basis for this report. '■. 

Two types. of investigations h^ve been proposed for research 
on the AWAVS simulator. These will be referred to as 
"performance"' experiments and "transfer V experiments , A "per- r 
formance" experiment is one that measures ope,^ator/sys-tem 
performance under ona set of conditions^ presumably uninfluenced 
by any other conditions. Measuring, pilot performance- in an 
aircraft under di-fferen t * instrument^ conditions or in a simulator 
with different configurations could be an exarnple of this type 
of experiment. A "transfer" study is one in whi(!:h the interest 
IS in the residual effect that practice on one set of conditions 
Ws on the performance of a second ret of conditions which 
follows it in- time. In this report, two classes of transfer" 
experiments are def^ined, ::A- "real transfer" (referred to as 
"transfer")' expejfiment for the AWAVS task is one in which the 
training occurs in the siraulator while the test of residual 
transfer occurs in flight in an aircraft. A "quasi-transf er " 
experiment for the AWAVS task is one in which both pilot training 
arid transfer testing (representing flight) occurs in the 
simulator, - . • 

Previous v/ork on this program had e?nphasized the planning 
of the performance experiments, the type to be performed first 
when the AWAVS si.nulator is operational. In this repqrt, more 
empnasis is placer^ on developing new and economical ' ways in 

t^^nsfejr^ experi.ments might be performed, to enhance the 1 
pragmat.i(S value of results from such experiments. | 

■ This report is not a review of the literature.. Its purpose 
is, to increase the understanding of .tlJose less familiar with 
"advanced expe;:imental methodologies" as they might be applied 
to the AWAVS program. It will aJ.so briefly summarize the con- ' 
ceptualization of new, ecoiiomi'cal approaches that might be ' 
employed to aid in the understanding and measure of transfer 
of training for the carrier-landing task. Detailed explana- 
tions will be ayoic^ed here. For a background in "advanced 
experimeflta;l methodologies," the reader may wish to refer to 
reports prepared by Simon (1970 through 1977). 



NAVTRABQUIPCEN 77^^:f0065-l 

• ■ . . . . ' ' ' 

The [oliowany tDpi'cs. wiii/to treated: in 'jihis. iN^port : 

a. A multif actor philos'ophy for AVIAVS experiments 

b. • AWAVS performance simulator experiments 

c. Refining economical multif actor designs " • 

* Applying economical miil.tifactpr designs to 

AWAVS per forifiance .experiments — an example 

e- Quasi-transf er e'xperi.ndiitrs 

f» Economic a ]..da,t a collection plans for transfer/ 
of trainirrg in the AWAVS simulator 

c • * 

g. Some unfinished business - measurement and 
'-^criteria. ' 
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SECTION II 

A MULTIFACTOR PHILOSOPHY TOR AWAVS EXPERIMENTS 

- - • . ' . . ^ 

« ** 

The. philosophy of AWAV^ experiments differs from that ' 
employed in other training simulation design and transfer of 
training experiments. For the AWAVS studies, a "holistic"' 
philosophy has been ac-:;ippted as categorically imperative. 
Th4s philosophy espouses the need to include in experiments, ' 

. during the factor identification and .function development 
. phases of a research program, as many factors as" ^possible that" 
are believed Critical >-o the particular operatidnal tasx under " 
investigation. The more one is 'able to -aohieve this goal, 

. the less- likely the da.ta w.ill be biased,' the more accurgt-ly ' 
laboratory data will predict the operational situation, and 
the more readily a quantitative, modular data base for 
applicaticr. " to future problems^ can* be built (Simon , -1977b) . 
Until -^attention was focused on .4:he various techhi^jues and 
paradicims for conducting systematically controlled- -larg-e scale; 
multif actor experiments, economically, the size of the effort 
was a limiting feature to this hp-listic philosophy. The 

. general approach tfiat is proposed for the AWAVS " experiments 
makes this no longer a critical consideration. 

Xhe novelty of the proposed approach lies primarily in 
the. econoniical patterns — both' spatial and temporal — employed 
m selecting the points forming the sintu],ation space that 
corresponit; to those in an operation'a^^l situation! Advanced; 
, techniques are also used to keep the infor-mation of primary 
interest unconfounded with effects- from irrelevanV sources and' 
to do so without dij^rupting the economy of the effort. The 
quantity and quality of. information -from this -multifactor 
approach in almost every respect exceeds tha-t obtained by 
other techniques- used by psychologists employing the same' - 
amount of resources. .' „ • 

"THE REDUCTIONISTS ' . „ ' . • ^ - 

That some do not fully adhere to this philosophy in the 
conduct of behavioral research is evidenced by a report recently 
prepared by a working group of the 'Vision Committee of the 
- NAS-NRC.^^ For pilot tiaining research at Williams Air .Force 
Base, this group recommended ways "to increase the .effective- 
ness of experiments" on visual cues in flight ..simulators . Their \ 
number one recommendation was: , ^ ' . ■ - \ 

' ^ • Simplify .the experimental design whenever s ' ' 
• possible. Attempt to i'dentify the major ' ' 

parameters with exploratory studies and'-. ^ 
then exarnine these parameters one .at a, J - .- 

time rather than .using a uultifactor design. 
• ♦ (NAS-tJRC, 1976,. p. 9) ■ ' • ^. 
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Earlier in^ that report they had lirfted a ?pumber of parameters 
that should he investigated in an initial evaluation of'tjie 
realism issue, and.'concluded that "the interactions between 
majdr parameters 'should also be studied but only at a later- 
date after the effects upon task-specif ic training have been 
determined by .varying one parameter at a time J" 

The report was in draft form "when it was seen and efforts 
to find a final copy have been* unsuccessful . However, the 
issue here is not with the report per se; it is mentioned 
only to illustrate the ^act :hat the one-f actor-at-a-time 
approach to behavioral research still has its adhej-ents; even 
among prestigious groups with considerable . influence* on the 
nature of 'major research programs. Consequently, th^ relative 
me'rifcs of sinqle. and multifactor approciches must be examined. 

SINGLE VERSUS MULTIFACTOR APPROAGir i 

. In the remainder of this sec^tioh, a comparison of two 
approaches will be Viade qis they' are applied to the task of / 
identifying critical, factors , and measuring their effects, and 
deri\^ing an equation to predict performance under operational 
conditions. A candidate Irs^ of .twelve fac^tbrs will be used 
to . illustrate how the information/cost ratio is a£fecteyd by 
experiments employing each approach "Cost*-* 'here refers to 
data collection cost. . ' " 

.In thi's di'SQussion,. the following claims will be supported 

Given the. same tiirie and resources, the multi-^ " 
factor approach will' always provide . ^ . 
' ~ qaantitatively and qualitatively better* 

information^ -fhan a* ^ingieV.(or few) factor 
approach will. ' • ' 

' ' ' ; b. There- is certain inf orjnciation that "a single . . 

factor approach can ' never pxovide; but which - 

is available when a mial4:i factor ' approach, is used . 

* Jl . t. 'J n ' 

' Information is judged "better" when*' it has more of the ' 

following ^^qualities : . " , ' ■ 

. . . ' » . « 

economy in data collection , ^ ' c 

- prafcise estimates within ^experiment . 

" accuracy when • predic-bing 'from iabora^ * ' 

" . .operational situ'^ion- . . . ^, * ' 

- ability , to 'g^ener^ilize to numerous, situations 

ability to use the data . to .construct, a modular * ^ . 

data base for future-reference , • r 

-'ease of spcN:ting| faulty data ' - . ' ' 

. . - reduced ^ambiguity in- interpretation ' . • ' ' • 
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Definitions . ' 5 

The number of factors in an ::f)Gri,ment: con range from 
1 to References in this re 3 or multifactor . 

experiments the>-efore are. as- opposite ends of 

that continuum. Traditiona iorcjl research in 

equipiiient design has incluc . three factors in^ 

a single experiment, and pri'. ase of "economical multi- 

factor designs , " ah experiment with more than five factors was - 

rarity (Simon, 1976b). In this section reference to a 
single factor or one factor experiment implies a class of 
experijnents rap^mmended by reductionis1;s who 'believe, that good 
\ behavioral :j^|tormation can be obtained Ih^y studying one factor 
_\ ^ t a time most, comments jmade J.ere_ ^§a§¥;ding^_ thi s;„ 

" .'VG'TS^ of "expeTilSents fwil^^^^ sometiiTies^'^apViyt^ to'V lesser degree"^, 

^to . experiments' involving two, three, four,\and even- five 
i_::-.:-£^±or:^H^ ,when--a— grea-i-^ 

. for the particular task under operational cpriditioris, ^ ; ' - 
■Reference to a "multifactor" experiment implies that it"^'' 
entails an effort to include most of the -^candidate factors 
'believed, to influence the- behavior found.- in, the particular . 
• investigation, \ Merely inclucling more than one factor i'n an- 
experiment would not meet- the requirement' of a multifactor 
experiment as the term is used here, . - : 

/ , . In practice,., there are. usually only a rela:H:ively few 
^ highly critical factors affecting per formance on a particular 
task.'^ However,- to include most of the critical factors in an . 

• experiment, it is usually, necess^ary to- start with a much larger 
„..?JiI^^ f ac tors .1 1 .ia,/as sumed - tha t - f or -mo s t ^ > - ~ / - • - - 

■ .behavioral problems, pe^rsons working in the field can identify 
candidate factors that have the potential for influencing the 
class of behavior under investigation, but . that for any 
particular fask, only an empirical effort can determine how 
much e.ffect each factor has, and therefore which ones are 
critical,^ While we may never achieve a one-hundred, percent 
inclusion of critical variables in a controlled experiment ^ we 
can at least increat^e considerably the number - over ' that which 
tends to. be typical in (^periments today. • 

An Illustfarativ e Problem ' 

Let us iook at one of the experiments that might be done 
on the.^WAVS program and compare what would happen if a single 
factor or a multifactor approach were used. Let us assume' 

• the^'e are twfelve simulator factors ' plus one ' pilot and' one task 
difficurty. (environment) . factor , all at two levels each. /Por 
the time being , we *shall not include 'the last two, since that' 

-would only complicate the discussion without altering the 
conclusions. The purpose, of jthe experi men t is to find out which 

■ ••of the twelve factors will be critical in the design of a pilot 

training simulator (u^ing simulator performance as the criterion) 
and what performance levels each of the two conditions (levels) 
.of each factor yiplds'. . , . 



^ NA>^TRAEQUIPCEN 77-0-0065-1; 



. SINGLE FACTOR DESIGN ^ ■ 

. The typical single factor approach might follow this 

desicy^u • Select^ one factor — <. Factor A and test eight 
pilots on the one condition of Factor A and eight other pilots 
on , the other condition of Factor A/. Pilots would be assigned 

- to. each group at random^ The re.i.aini ^ ol-^v^ 'nnilator factors 
would be held constant as, prer^: niiab i ^ im<: ijirelevant 
sources of variance. This desigr. i i r I in Table^l*. 

When the data has been collected the mean performance for. 
each; of the two conditions (levels) of Factor A can be calcu- 
lated and the effect of Factor A, i.e., the difference betv^^ . 
these tw6"meTns, can be estimated. The precision with which 

"^each^f fect'^T:s'"'e^^ 
difference {o^'^) , can also be calculated. The equation for this 

^ip3dAi-stea-ti^(i^ - ----- 




^ . oi^ ^ 2a' 



= . 5a 



where a^, is the estimated error variance of the experimental, 
unit (independent o.f factors) , and. N f& the total number of 
observations made per experimental condition.. Once" the appro- 
priate a is. established, this standard error of the mean, 
difference can be used to set confidence limits about the - 
empirically deter.nined means. 



MULTIFACTOR DESIGN 

. Using the . muj.tif actor approach , the effects of all twelve 
factdrs would, be estimated in a single experiment also composed 
of a total of 16 observations from 16 pilots, one per observation. 



Slightly, modified experimental desic^ns have b^en used in 
"one" facto.r" experiments. For example, a .subject (pilot) might 

;be tested on both experimental conditions . To compensate for , 
carry,-over effects,, one-half the subjects would be presented 
th^ conditions in one 'order, and. one-rhalf in the opposite brder. 

. l^or pur discussion, these, variations are not critical. While 
only eight subjects would be required, the total number of 

" observations remains 16, and it- is the number of observations , 
not pilots, thcit will be the unit of measuring the cost effec- 

. tiveness of the data collection. 
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TABLE !• EXPERIMENTAL DESIGN TYPICAL OF. 
ONE-FACTOR EXPTilRIMENT 







^ FACTOR A 




FACTORS HELD CONSTANT 






Condition 1 (-) 


:<!i tion 2 ( + ) - 




Factor 


Value* 








1 


9 




B 


CO 








2 

3 


10 
11 




C 
D 


u 

o c 
m QJ 

-OJ 






Pilots 


- 5- 


12 

- ^ 1^ - ■ - 




E 


•H 

u u 

O QJ 

v; — 








. 7 . 
8 . 


14 • ; ■ 

■ ■■>' 16. 




"g 

H. 
; I • 

J . 
. K 

L 


(Single vaJ.ue 
selected at e 
option . ) 





* - . ■ 

J. ^5^.-° 1th e_ other 

- or -f. With quantitative values, these would correspond to 
low or high levels, and be a shortened notation of -1 and -fl. 
The valuer at which each factor is held constant would be 
decided by the investi^-^ator . ' tj. 



- The experimental design for this ^2 experiment is shown 

. iTi Tabl^ 2-A. The 4niri\is dnd plus signs in the. table represent 

the high\^or low (or first or s econd) level of each f act o r. 

E^ch-TTow- r epre s en t s a d t e r €$nfc experimental condition and each 
column— - up. to twelve — - a different factor. With this" design, 
the main effects of all twelve^ factors can be estimated. The 
precision with which each one of the main effects can be esti- 
mated with 'this design is the same as the precision of the effect 
estimates in the single factor study, namely .5a*. Thqs , finding 
• the main ;ef fects of twelve factors with- the single factor 
approach would cost twelve times .as much a3 With the multifactor 



In this example, the multifactor .design is ' not replicated ; 
^therefore, there is no direct estimate of the "error" standard 
^devxation ' (Oq) : Internal estimates can be made, however^ from 
the half -normal plot as shown on page 44 (see Simon, .1977', .p 97) / 
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TABLE 2 . 



MEAN 



DL 

CK 
BJ 
EG 



HL 
FK 
AJ 
EI 
B 



TYPICAL MULTIFACTOR EXPERIMENTAI. 
OKSIGN (FOR TWELVE FACTORS) 

Main Effects and Aliased Interactions* 

IL AL HI GL FL BL CL fel DI AD 

Ak IK CH BK HK GK DK. DH GH CI 

FJ HJ AG CJ IJ DJ GJ CF BF BH 

EH EF DF DE . AE CE BE AB AC FG 

C D E/F G H I . J K L 
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^* ' In the first block,, main effects 'are aliased with two-factor interactions : 
as shown, "alonq.with hiqhor-or,der interactions. Where, no malti effects are shown, 
one of a string of throe-factor interactions, is shown in parentheses. 

.. When data from the second blockv-ls added to that from the ' first block, • 
main and tl^ree- fadtor interaction off ects are isolated from the . strings of ' 
two-factor interactions , . . , 



■ "...Block 1 and .Block II are each. Resolution III designs, 
form a Resolution IV 'design; . 



Combined, they 
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.^study and the preciioion of eoch c-'Stim/i Lc: would be no greater. 
"Conversely, the. main effects of twelve factors can be estimated 
at the same-^data collection cost with the multifactor approach 
' as one can. estimate one factor with the single factor approach, 
and with the same precision. 

Mean Performance Measures 

There are still more important and" subtler differences 
between .4:he two approaches that are often overlooked. For 
example; the means obtained with the single factor study/will 
be different from the meanr^ obtained in the multifactor study . 
."'This^is so in spite o^ M tact that both were obtained by 
• 'measuring ^erformanc' .yh times at th''^ ' ' 1 (of Factor 

A) and eight times at the low level (of 1 actor A). Unfoirtu- 
nately, the means obtained from the -single factor study are not 
V representative of performance throughout the, experimental 
=^v-i^:F<^^^»-— rl=r^^tead= -abira±ned"hy~nie asur ing~o"nly 

t.w.G- locations out 'of a possible 4096 in the total experimental 

• space (in this example) . These two locations . are at the . edge 
,: ■ of the twelve-dimensional hypercube, representing less than 

five, ten-thousandths of the full factorial space . , 

'V .^^t it is-not the small proportion that is critical, per ' 
se_; it is the fact that these means estimate are" not in'^.epen- 
• dent of the factors held constant. In spite of many replications 

• ^^^d/^h^t might appear to:be a very uncomplicated experimental 

■ '^\:design,: ^the chances of obtaining a reasonably accurate estimate . 
. of .the performance on, either, the high- or *l6w condition iof ^ 
Fac.tor A (in our example) is .very .pdpr. when the single factor^ 

apprx)kch._..is._„used^_ This-is becausfe . the^- 'answers, we obtain w;Lth 

. such, a^ design depend on which^' values the ijivestigatof * c^'ecides 
to use for 'the factors held constants Because* they are 'held 
. * constant doe£5 not ftiean that they hav'e no., effect on per f ormafice\" 
they do .' - . ' ■ ^ . ' "" --^ • . .'. ' • 

■ If. a factor that is held constant . would critically affect 
performance were, it varied, then the value at which it is ^held 
constant will make the overall , task either easier or more ' 

; difficult, to perfornu Thus/ mean performance on^the conditions 
of -Fac tor A .would _ i3XCA_e_aRe„.or _decrease froTTi the average/ de-" 
pending on the particular yaiues at* which the constant-factors 
are fixed by the investigator. In the singjl^;^ factor study, 

' y the combir\ation of ""fixed values is only one 'out Qf a possible 
:2048 alternatives (i.e^., one. out^ of 2 1 1 combinations).. Since- 
the single factor experiment tells us nothing of the effects of 
• these factors , .we have no way of knowings in which "direction .. 
the bias lies nor its magnit\ide. . ' . 

■ ■ • " " «> . . ■ ' ■ • ' .... • 

With a multifactor -af^proach, the situation is different. 
..The means for Factor A are more, representative since. they are 
obtained by sampling a nuitiber 'of conditions throughout the 
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experimental space. The other factors are not held constant 
but are varied systemcitically and orthogonally to one another 
as well as to Factor A (see Table 2-a) . Each level "of Factor 
A is measured in combination with an equal number of high and 
low conditions of every other factor/ thereby neutralizing the 
effects of the other factor on ,the mean performance for each 
condition of Factor A,. The same balance occurs with all other 
factors' in the multif actor ex[)eriinent . 

Graphic example . The' above relationships may be mo^e- 
easily understood if they are shown graphically- Since it is 
difficult to draw, a twelve-dimensional space on two-dimensional 
paper/, let us use ,a five-dimensional space to illustrate what" 
has been said so far. " In Figure 1, two diagrams each repre- 
senting a five-dimensional space are shown The one on the 
left will be used in the discussion of the single factor desig'n 

and the one on the right, of the multif actor Wesign. In_the 

diagram on ' the lef tT^t~the corne3rs^*~of~each. cube , the thirty -"two 
conditions of ^ five, factor,. two levels per factor space, are 
identified. The .conditions would be identically named in the ^ 
corr&'spondincj positions on the right. ' The "^conventional symbology 
•for naming experinVental condLtions. is employed, where the ^ ^ 
presencei^of a letter, a through e, indicates thBt the high ( + ) 
lc\/el .of factors A through E respectively is represented. The 
absence of a particular letter indicates that thd low level 
of that factor is repres'qnted in the condition . r Black dots 
have'b.een imposed on each .diagram where data is to be collected. 

In the single - factor experiment , two conditions ,at which 
performance 'under the- high and low levels of . Factor A are to. 

be cdmpared are selected arbitrarilyT~i~.evT"b£"^^^ 

that in this five factor .case, ,:any one. of 16, alternatives could 
have been chosen, all of which run only, along- a horizontal edge 
of\,a cube, in Figure .1 . On^re' the two conditions ^are "chosen, 
eight ; measurements are made at each condition. Howeyer, if. 
Factor 'C has a large, effect pn\performance with the*+ condi-- 
tion causing the higher performance level , then the means at. 
be and abc would be . higher than if the ^.ingle factor study of 
F^jactor A had beQn carried out with Factor C being held constant 
cit vi't.s;- lower level.,, e . g . , conditions e and ae . This process. 

become.s; even more complex i-:^-other e.onS"ban1^--5^ ~ 

critical effects . Even after data has; been collected through 
a series of single .factor, experiments on ali the factors. 



Any pair of conditions could have been se^^ected as long 
as a is .absent f^om.one and present in the other and all • other 
letters are held constant,' i.e., the same in both. For example, 
bee and abcc, e and ae, and so forth, could' also have been used. 
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Fiqure 1^ Location of Data Collection' Poinjis (•) 
. ■ '-' In A Five Dimensional Space. o 



Each point in the Single Fq.ctor. study is replicated, eight times. 
This makes a total of 16 obseiyations. In the multifactor studyv the 
16 obsorvati^ns arc di^stributod as shown with no' replication . ^ 



there is siill' insufficient information, to correct the mean 
estimates for potential biases. The fact that the individual' 
onej factor studies- were per formed sequentially without any. method 
of correcting or measuring possible sequential .effects that would^ 
cause irrelevant variations in performance from study^, to study 

srfbrmance even^more suspect. .. 



makes any o^stimates^ of mean pe: 



, ■ j - In the- mi uti factor experiment the data points ^ (shown In 
Figure. 1) were selected to prevent the mean performance values 
from be>ing affected by the other factors in the experiment. 
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The cdqht-. points marked "minus" rcprcjsent the low condition of 
Factor A and the e^ight points inark'Gd "plus" represent the high 
coriditioii of Factor A. Means oC tho measures obtained for the 
minuses and. for the pluses .would represent the averages for the 
two levels of Factor A. Note that half of. the low level points 
for. /Factor A were measured under a' hi^^n condition of every - 
other . factor and half were mepisured under a low condition of 
every other factor. Any effect that these other factors might 
have on task difficulty has been balanced out in tho rvH in^a/ 
of factor A in "this multifactor p.lnn, sar.v v/vju L^- true 

wiM'o tVv moan- \\\y 01 u r , ; lors oh^timated. The main 

cllcci^. I (-^^.li are orthogonal to one another* • 

• Interaction Effects 

• With no interactions among the factors, even the single 

factor approach will. arrive at an appropriate estimate of the 
: effect of F'actor A, This is- true even though .the means of each 
condition, as previously illustrated , may be higher or lower 
than what their. "true" value would be because the factors held 
constant are at.valu.es thafmake performance easier oqr more 
'difficult,.. When_there is no interaction, since both means are 
^-^ajffectecl the same., the diff e.rence , between them would remain 
.H::onstant whatever "th6^ effect of a fixed factor. \ 

For exampie,, in a , single factor study, if eight, measures 
^ were, taken eaqh. on~^ the^jiigh and the Idw^ condition of. Factor A, 
and . if . Factors. B and. C are^each^ held constant; at their high 
level and Factors D and E, at the^^ow^^LeveL — the data col- 
lection * points be and abc indicated . in Fi^ure^-l^rir:^ the fpllowing 
fictitious data^ might be obtained: . . ~~^T~^---^ — ■ 

•"^MEANS. OF GONDITIONS;: be == 23, abc = 32; ^ 

■ . EFFECT OF FACTOR A = ' • \ 

,If , instead, the! high and low conditions .o.f Faqtor A were com- . ' 
paired when "only Factor B was h'eld' Constant at the high level 
arid Factors C, .D, and E we.re each held, constant at th'ei-r low 
level, and ' if -Factor C actually had a. strong effect on performance 



One alternate plan exists for this example, • The undotted 
points might have been ..used, instead of the dotted - points with . 
the same results-, ' This happens' to be a '2^^ fractional fa.ctoriai. 
Noie- iha-t points -in- this rdesign are always located ' on diagonals , 
in contrast to the horizontal location of points in. the single 
factor experiment , 
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e.g. / ^2, then a change from the high coiidition to the low 
condition of Factor C when it is held cqhstant, would cause the 
performance scores' to change by .-4 points, i.e.- 

MEANi5 OF CONDITIONS: b V 



The means dropped, b^ut .in the abs^ce of interaction effects., the 
ef^fect of Factor A is unchanged.. 

Factorial Effects in the Presence of Interactions . ... 

But. if there are interactions then we cannot trust: the 
estimates of the' effects in either the single or multif actor • 
approach. However, there is "a^^^^^ 
..|:he multifactor approach can handle "this problem while the .single 
factor appyoa[GjV:::e3gLfija.^^ 

interact with Factor A such that when the high lev.eis of both . 
occur in the same experimental. condition, performance is im- 
■proved far beyond what would be expected from a linear combina- • 
'tibn of the ef fects_pf us . 

say...it adds, nine poin^ts to th:e mean of that eonditioh • 

In a single facto :: experiment ; we might get these- results- 
if^ Factor A were studied with. Factors B and C held :coristant 
each at their high levels and Factors D and E at their low ' 
levels,, e.g., the; marked conditions in Figure 1: . . ;? 

J..- MEANS OF CONDITIONS: be = 23, abc = 32 ' . ^ . 



/ EFFECT OF FACTOR A = -f9 - 

.■ ■ < '■ .. ' ^ ' . ' 

But. if the investigator had by chance chosen to hold Factor B " 
at the, high level and Factors C, D , c and E at -the Ibw levels, *, 
•these results might have been observed: 

MEANS OF CONDITIONS-:; b:=-- 23, ab = 23 . 

'EFFECT OF FACTOR A, = 0. " , . ~" ., ; 

Th g TiTrEeracTi"on effect, when The hrgh'"'c'onditions' of Factors A ^ 7 
and C occurred in- the same experimental condition together, ' 

Uas in condition, abc) , made .Pactor "A appeat to be a critical , 
^^ect. But' had the investigator used, conditions' b and ab ■ '. 
wi^K^Factor C held constant at its low' level, the results~would 

. have to the conclusion that Factor A was .not an important 
factor, I4i equipment design. Thu& he might decide to omit it 
later-in:^ interaction study., .in the study with 12 factors' there 
are 2048 p^ars of points to choose' fr6m/ Qf_which. if— Factor C 
-.o'^^^-^ — one--set of ia2 4 would hi^e ded to 

■the conclusion ti^at Factor. A , was trivial and the 'other 1024 tb 
the .Conclusion thaXFactor A was critical. . A 50-50 chance, some ■ ■ 
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may fec?l, are good :> riL ox[ rs 0< not Ly 

on ch.\: ce The onl \ t two ' ^ Oi nteract: i 

, is to uu 'ude both facioi:;; -i; f.iu' cxperimen ^ . 

In a multif actor experiment , the situation is somewhat 
different. It is true that with the design shown in Table 2^A. 
\. wherein 12 main effects can be estimated from only 16 experi- • 
mental conditions/ all main effects are completely confounded 
with some two factpr and higher order interactions and suffer 
the -^same ambiguities" of interpretation .as in the single factor 
, experiment, However, by adding oh'ly 16 additional experimental 
conditrons to the design (see Table 2-3)/. yje isolate the 
twelve main effects from all two ^factor interaction effects.^. 
Although We have doubled our original allotment ° of only six- 
teen observations to achi eve this, we have still usdd only one- 
sixth the effort required to study' all twelve factors with a 
single factor approach , What is more., in the single factor 

still remain .confounded.. ' , ./ 

Summary of Costs and Benefits - . . 

! , Let ^ us summariz.e v/hat has bGen foUnd out regarding the two 

approaches up to this point • What are the costs and benefits 
\ of using each approach — single arid multifactor to determine 
thfe relative importance o£ twelve factors? To achiev.e- this , ' it 
is necessary, as a^minimum, to determine the eff.ect of eacH 
:^actor, isplate'd from two factor interaction effects , but with 
critical interactions . identified^.,^ Table 3 summarizes costs, and 
-achievements, described up to.;thi^|^point.* . .. . c -. ^ ' ^ 

' - • Therefor^, wi thJ.32- observatl.oTrs'^TnrTlie multifactfor experi- 
ments we can study the\ef.fect:s of 12 factars' with even greater . 
precision than we would-have obtained "with 19 2 observatr 
xequire.d. in the series of single ^^ctor experiments.. ' F.urther- 
•more, for^.each new factor to be studi;pd using the single' factor- 
approach," another increment of ' obsef^^aticjns are required , in • 
our example, ^an additional 16. Using.'this multifactor design; 
this /is not €he case,,- To isolate the iRiain f-rom«.all two factor 
interactions , we' can. ^ study up-:- to- 1'6 ' f-acl^prs with 32 observations , 

nn mnrpx. fh^^n w pr^^ r<=>qin r-pr! ' to .Ql-nHy ^ 7 .1 ^ Tt wo ^ ll d -rpqn i rp n n l y __- 

•64 /^Observations still f^wer than the number . required > to study ." 
12/factors by the single factor .approach --ilto study* up to 30" 
■ factors with, main effects isolated froiri tw6:' factor interactions!. 
. . . . ■ ^ • " ■ ; ^ '\ ■ - ■ " . ' ' * • 

\ in .bo:th approaches, main'effects can be biased by . 
three factor xnteractigns . More .data must be collecl^ed _ to.:_. 
isolate these, if necessary. ,., when . mul-ti-factor"'>aesigns'are .. 
• employedi- . No; r.eco'urse is possible .with the- single factor study ,^. v 
' which ^cari only start *pver again without clues — and dp a 
. ■ multifactor study to. discover and isolate interactlon3 . 'The • 
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TABLE 3. COMPARING SINGLE FACTOR AND 
. MDLTIFACTOR APPROACHES 



' Single Factor Multifactor 



Number of main effects isolated 
from one another 



12 



12 



M^ein estimat^es of experimental 
conditions . are unaffected 
-by level of other facitors 

_ i-f~-t-hexe^- ar^e'-^niDT'-xn^^ ractions 



No 



Yes 



Precision with which each effect 
is estimated* 



• 50a 



• 35a 



Number" of "main e"f^ect:s isolated ' 
, - ' fr^m two -factor interactions 



.12 



Estimates of main effects are 
- _ affected if. two factor 
intieractionsr-axe present 



Yes 



No 



"Detect thev preserice of two 

factor interactions or clues' 

_a§_^:tO-_wJiete_JtwbJ-f actor-^^.^ 

actions /mi^ht ^xist . ' - 



Main effects, confounded with three 
factor' interactions 



-No 



Yes 



Yes 



Yes V 



Total number .pf observations 



u s ed to a cRa e ve 1?fii s 



192 



32 



.Planned capaciti^• to ^ further _expand- 
^ ^ experim_en-tal- apace" by. aug~ 
.• .menting existing data 



No" 



Yes 



. Each 'ef fect in each single f actor 'experiment was estimated 
with 16 observations. Each effect in the augmented multifactor- 
experiment was estimated with 32 observations. 
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multif actor approach builds on the original data. Since the 
original single factor studies can be biased for the reasons 
. cited earlier, they cannot even suppiy data that might reduce • 
the. size of subsequent multi fac tor studies , i.e., the number 

• of factors nee4ed to be included in the multif actor studies.- 
It would be risky to eliminate a factor based on single factor 
study information. 

Without replicating the experimental designs, an act that 
•would reduce its. economical quality.^ the multif actor approach 
has no direct method ot measuriiiq^ error variance , and there- 
fore cannot make a traditional test of statistical ^significance 
. of the differences. VJith the single factor approach , the 
wi thin-cell subject variability is conveniently labelled 

. "err or" v ;=>r"'^n<^e ^^r! fh^^ m^nhrinins of a tes^t o :i;^ai;.^..ti sjtica 1 

^ significance can be followed. This does not red,ace the effec- . 

tiyeness^^of . tiie multi..f^q^g^^^^pr^ 
^-rrea^onsT For^c^ the test - of . statis t±i:ral7s 

• as it jvould be-appl.ied" here is . of limfted value in the int^rp- 

■ retation of the ddta .(Simon, 197.3; 1977.b; NTEC, 1976). '^Fbr^ - , 
; another, there are. other equally effective methods of examining 

— - wh e the r"" P t s ex ve d e f f 

^the- mu'lti factor approach as used One of these , i - e .,- half- . ^ : . 
normal plots, is. illustrat:ed in Section V. Economical partial 
replication 'techniques are also available.. 

- .i £^t:her Consideratio ns . ^ . . , ' 

The.re 'are less tangible but equally important reasons 'for . 

consi dering ptily. a mult i f acto r . approach in equipment design ._ : : 

research ., When a multifactor approach is used arid we have 
shbwh^tKat it is much, ipore economical .—r- . the informairon^ 

„.^„„ob'taine.d w5.11- be more generaLizable, --will i explain more , -will - be' - '-'^ 

easier to interpret, and will enable .more accurate predictions ' 
to be mac^e from^ the laboratory * data, to operation -situations-r^^ . 

Generalizability . Multif actoriLappro'aches are more gener- - 
alizable by ,the very •faci:--^hai::^they investigate^. m conditions of 
more facjtors-- G.iv^'fKthe results' from one ■ of these' exip'efiments.,- 
_,-^-aia— xTTyesti gator may. ' consider a wide ra nge of a I ternati ve si mu- • 
"~T lator parameters; to be- truly generalizable , the' experiment must 
also include contextxial fjictors.. ,For example, pilot training, 

■ simulator studies have soijietimes been critized becayse 1:hey used 
pilots with one kind of experience to obtain data that was. 
applied to situqitions in which the pilots have different kinds .. 

• bf experi^ences ; or in studies done under simulated conditions ' 
-for low perf^prmance aircraft when the results were applied' to . 
.situations.' in which high performance aircraft would . be ^involved- 

• While this is ipore . the f atilt of . the user than . of the experi- 
menter, still it .raises the question of' whether or not non- 
representative experimients can'be justified at all? Simple . 
.experiments, lack generalizability; , multifactor experiments can 
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achieve' more generalizability by including not on)./^the simula- 
tor parameters, but others associated with pilot, task,, and 
environment characteristics. If introduced ■:t the bec^innirig of 
the .research program, during factor identification stage, they' 
. can be studied far more economically and enable more generalize 
able results to be obtained, ^ ^ ; 

* • . . 

Component contributions . The multif actor approach can 
also provide ^better information than' the single factor approach 
" -in situations where complex devices are being studied, ^as in 
the case of a pilot training simulator, " While one, may think 
of: tfee. visual or motion system as; . unitary components.^ results 
■ maiy be totally misleading when components as . complex as \:hese 
are treated as units/ Each is. made up 'of sub-components which 
have their own individual effects on performance or on transfer 
~ of tf a in 1 n^77"'~7r.lTiioTi o n / noT^nio t i dn ™ stlTdy point • / ' 

• Motion in a simulator can serve two relatively . diverse purgoses: 
^ •^-^^H=^ i "t:rc a - i^^ 

'• better control his aircraft; or 2)- it can simulate environmental 
disturbances that can negatively. affect the ease with which the 
ciircraft can be controlled, S.imulatiYig. these two purposes may. 

::_not have, the: same -A" study^ in Vhlch. these 

effects aire riot examined separately, as two. independent factors^ 
.might lead to the conclusion that there is no overalJr^if fer- 
ence between a motion' or a no motion system, if th^ effects of 
these' two components were in fact in opposition and cancelled one 
another: A simi.lar illustration might be ^ used in regard to the . . 

' * study of a motion -system in which several motion cues are used,". 
« ;e.„g, , simulator movement and G-seat&. Unless they are studied 

^/^ "^ separately (and the multifactor approach is the cheaper way. pf « 

doing this) , their effec ts mi ght cancel one , a nother-._ SimilarL^^- 
ac ebmparison of .'two simulator configurations to see which is th.e 
' better might- suffer from -this same problem, e.lgv', the existahce 

:„::.::ro"f---^^ system in one corifigurat'ion; and a superb- 

motion system in the other, configuration might lead to a stand- v 
Off, showing both to be similar in effectiveness and never 
revealing which combination might have produced the supex- - 

'1 . sim.ulator so long sought after • . : . . ' 

Interpretation , t When^ only-- two data points are investigated,- 
the inv e stiqatot ha s no way of evaluati ng tHe correctness of the 
results through rational processes. When a great many data 
points are collected^ in tKe . systemaitic '^manner of ' th multifactpr 
designs, the' investigator has built-in checks in the form of 
' data patterns . Erratic -^behavior is more likely to be spotted, „ 
^ qj-Ving the investigator the opportunity " of checking whether it 
•is an outlier or a bona fide interaction-. t ' ■ 

A multif actor approach 'also, puts the interpretation of 
^ - experimental 'results 'in perspective. When a • single factor ^is " 
studied dlone, it is more difficult to jucjge its relative .im- 
portande to the. system. .Importance is more .clearly .j^viderit ; 



^ "naVTRaVqUIPCF-:N 7 7-C-00G5-.1 ' ■ • 

•• • ■ ' - . 

•when the proportion of variance ^i facMor acic^ouiil-.s for i^s known 
relative to that accoun ted. f or by . a L 1 of the other orimary 
factors affecting''''a particular ta^sk. When allocations of titne 
4nd"nioney require that improveriven Is >n' ecjuipment design be 
cons?:ldered on 'a priority . basis , knowledge of on'e factor^s effect 
on systein performance in context with ail others -is an imoortant 
interpretatifve feature provided l|est -by the multifactor approach, 

Prediction > The experimental designs traditionally^ 
'■■"employed, by experimental psychologists have been more concerneci 
with precision of resul^tsc rather than aecurac;y. .Precision- 
refers; to the repeatability of a laeasurev whether it is biased 
(inaccurate) or not- The single factor appro'ach, as has.' been . 
shown / maximizes bias and obtains a satisfactory l^.vel of pre~ 

i s i on only at consid erable cos't > The multifacto r appr o ach 

^ (with., a holistic; phil^ emphasizes the "reduction bf^BTas- . 

and. ^t the same time , ^because of its inherent features , tends 

of the ^single versus miiltl factor . approach wi^re* dis-cu 
,nt"he eminent statistician ^/'Fra^^^ ■(1935> p-.5.); more .than . o 

forty* year-^ ago/: At that . he made' thp. following comment: 

- v» . experirneriter /.who conf ipes himself to ..Experiments, on 
• single factors'/ making a '^ues's at.' the: final Uevels of the other - 

factors,, merely etriulating tlie .tactics of an ostrich." 

Because we can include^ iq -our experiments mos^t pt the^ ; 
f actbrs crftiaaLl under?, operational oonditions , as well as those ' 
affecting the pilpt, task/ and environment, the* multifactor 
approach increases the accu^^acy of our predictions.. ^\fhen the 
'^single factor approach 'is used', each critical factor omitted . 
' (held* cohst^ni-) froTP an f^guation can bias a.predigtion if 



;does not . match. that Eound qpera tionally ; ,each; one that is ^ 
allowed to v.ar.y„ in" the experiment •results' in. .variable prjediction . ^ 
Terror. . Even at the end of the experimental program when only* a 

few. configurations might be* examined for purposes of veriffca- 
.^'tion, detail^' comparison, or for. establishing^ fiducial limits 
p.n the performance , the mulji factor approach has already ■ v:. 
provided an overall framework into which the data 'firom^the- 
lim'ited experiment can be anchor/ed. y . . - 

' " * • .. . ^ V ■ ■ . \ y.."'- 

' The use of g/'sequential block tec hhique for data collectio n. 



. in' the multi'factor approach can help optimize prediction. If 
; the investigator has re.cison .to suspect that, the order of his ^ 
^predictive model is inadequate, i.e.f would fail to; fit • reality , 
he nay collect additional data that would be .combined with the 
original data so as to enable quadratic or higher-or^.e^^ surfaces 
' to be estimated if necessary. . - • '. • 
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SOURCES OF ECONOMY IN MULTItACT.OR EXPERIMENTS 

Hdstoricaily , accepting the need tc perform holistic (multi- 
factor ) experiments has proven to be. easier said than done. In 
1954 ,:' for example,- Williams and Adelson, wishing to examine the 
effects of, 34 factors" th^y believed important in the design of a. 
pilot trainin^g .simulator ^ were stymied by the fact that a facto- 
rial design for 34 factors 'at txve levels each would^ require 

.5.8 ^ 10"^-^ feombinations 4^ Even studying each factor, one at a 
time, at five levels with all other factors held constant,, would 
have required 3400 observations • To reduce the effort th^y con- 
sidered studying only the important factors, but recommended 
that no study be done' at that time since the original 34 had 

"been selected because they wer.e the important, ones • The same 
quesitions regarding pilot training L^imulators and a method of, 
doing a comprehens*'ive experiment continue to exist during- the 
intervening' 2 5 years. Simon (1970a, b, 1971, 1973, 1974, 1977a, 
b). proposed a more economical approach with which to accomplish / 
thi^ task. . A f ey? of thet more ' important principles for achieving 
this economy are cited here brief lyi ^ 

First of all, it is not necessary to collect data with which, 
to visolate h^^gher-order interactions. In the example cited 
abpye, it is a -certainty that no 34-f actor interaction would be 
of any practical importance.' For that, matter, no ten-, or six-, 
and probably nd four factor interaction^ will have a practical 

•effect on performance. ' Even -three-factor interactions seldom 
have large effects, -particularly if quantitative, continuous 
factors are'involved (Simon, 1976b)., To illustrate the savings 
this observation can /achieve, let us consider a 15 factor, study. 
A qomplete factorial for 15, factors ^cwould require 32,76C combi- 
nartions if each factor were studied at two levels, or 14,348,907 
combinations if each were studied at three levels. However, if 
the response surface for 15 factors could be represented by a: 
first-degree equation , only 16 properly selected conditions would 
be required. If . it could' be represented by a second-degree 
equation, .then c>nly_ 136 conditions would be required* If the 

•surface could be represented by a third-degree equation, then 
816, conditi(ms would be required. While the latter number of 
conditions yCs Still l^rge, it is OQly a ,-00b057th fraction of 
the cqmple€e^ 3*^^ factorial. ^ ^ 

To be economical , however, an experiment would never be 
started with the intention of measuring 816 conditions ,: even' if 
we thought that a third-degree -surface need be represented We 
would begin' by collecting only enough data to approximate a 
first-degree surface. Tnen a little, additional" data would *-be 
collected in order to test whether this' first-degree "appro^jiimation 
adequately fit^^ thc^^ response surface. If it does, the 'study can 
stop, thereby achieving coiisiderable economy.^. If not', additional 
'data would be collected to approximate a secondrdegree surf ace..and 
second test would be ma'de. If the ■ fit is adequate the stuc^y would 
stop, at this point 7' if not, ^ it would continue. This iterative ^ 
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process serves two purposes: one, it keeps the cost of the ex- 
periment as low as possible; two, it provides the assurance that 
the response surface will,, be^ adequately represented'- Theoreti- 
cally, the procedure could continue up through fourth and fifth-^ 
degree • surfaces r although this is highly unlikely with psygholo- 
.gica.l data. Interactions at that level would more probably 
Indicate that the data were carelessly collected or that the 
experimenter had failed to scale his d£ita properly. 

Proper scaling is another v/ay to achieve e/::onomy in multi 

factor -experiments'. Certain classes of interactions and curva- 
TCrr^ can b,e eliminated by selecting the v appropriate scale. If 
care is taken before the data are *.collec ted* to select the correct 
scales, the necessity of approximating a third- or even a second- 
degree surface is diminished and. less data need be collected. 
Certain interactions cannot be. avoided by scaling, but in the 
behavioral sciences' these occu^rr infrequently. 

Still further e'c9nomy can be achievedif we separate the 
critical factor identification process from the function deriva- 
tion-process. Why should we collect the data required to de^^elop 
a third- or second-degree function for 15 factors if all L5 ; 
factors -are. not truly critical to the specific task under investi- 

^gation? In large scale experiments , we introduce candidate factors 
which rationally might be expected to be important to :the task but 
may not be . Our first goal is tQ determine empirically which 
really are important. A screening study for 15 factors can i)e 
designed requiring as few as 32 and probably hot more than 50 ob- 
servations to provide ^'thie *data needed -to order the factors accord- 
ing to; the ^magnitude of their effect ..on the perforpance of the 
specific task* The extra 18 obser^>ationfe are used to isolate- 
critical two-factor interactions • It is unlikely" that all 1.5 

'factors will be. important; in fact a good guess would ,be that 
fewer than half will have large practical effects^ In any ,.casp^ 
even, i^f only a few were,, eliminated by this screening process we 
have reduced, still further the; magnitude of the data collection 
process require to map the response surface. - Furthermore r the 

data required to develop the higher-:prde_r r^ 

a" t^st indicates it exists) are added in orthogonal blofcks to the 
data from the screening study, a^ savings which helps keep the data 
collection economical . ; ^ .. -^ 

Up to /this point, nothing has been mentioned about the expen- 
sive habit/of replicating complete designs. Still further ecpriomy 
is incurred when a multifactor study is performed by replicating 
only when it is necessary. In an earlier section of this report, 
it was • shown how "hiddei)" replication' provides adequate precision 
at considerable savings in data collection ; . The existence of 
trivial factors also provides an internal' source of degrees of 
freedom for estimating: an error vari.^nce if. such is required* 
Finally ,L 'techniques of partial replication can be employed — only 
selected conditions are repeated — - for an extern&l estimate of 
error with which confidence limits of the response surface can be 
calculated.^ 
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One final word about design economy. Because fewer data 
points are eollected, some information will be lost, P^resumably 
only the information the experimenter has determined is unimpor- 
tant" to the task. Still, in the absence of replication overkill 
^oui^d in traditional few--f actor studies, the opportunities for 
bias to creep into the experiment are' higher. Outside of .careless 
data collection and a failure to control irrelevant sources of 
varjtance, the inost common experiment-induced source of bias, in 

■'^psychblogical experiments comes from the need to collect data 

._se.queniu.aiLy--.~->--?^rend and trial- to-trial transfer effects are 
commonly found as a re/sult of equipment drift and operator learn- 
ing. In certain screening designs, there is a built-in protection 
against trend effects that requires no additional data* collection 
(as is needed in traditional experiments employing counter-balanced 
designs). As a result, one run-through of a sjLngle design is suf- 
ficent to isolate any trend effects from the effects of interest. 
Additional data are required when trial to trial transfer effects 

'occur or are anticipated. If subject characteristics critical to 
the -task are treated as factors in the multifactor "study, then in 

'many instances total design replication, to account for those 
"individual differences," is unnecessary. • ' 
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AWAVS PERFORMANCE SIMULATOR EXPERIMENTS 



; . When the AWAVS carrier landfhg simulator is made available 
for research, current plans are to perform, a number of . 
"pierformance experiments before actual transfer experiments are 
cpriducrted (refer to page 7 for definitions) . A brief 
desci:iptioh^^bf several of these are given here. 

GkOSS-EFFECT STUDY ^ - - 



; , A preliminary coiRparison would be made of carrier landing 
performance by high and low ski].l/exp^rience pilots on the 
"best" and the '*worst"' configurations of the simulator and 
und!ier two-levels of task difficulty. "Best" and "worst" in 
this case refer to the quality of the physical system, , 
particularly the visual scene and the motion system- 



This: would serve several purposes.' The information ob- 
.^ainedi i.e., the differences in performance under the best 
and ^orst available simulator, configurations, could influence 
futurp research plans/ For example, if the differences are 
quite\ small then one may reconsider conducting the full scale 
multifactor study to identify only s-ubtle effects of little 
practical impprtarice. While this single experiment would not 
-be sufficient to abandon a].l research, a small practical 
difference between best and worst ^conditions would certainly 
requilr^^ 'the investigators, to reevaluate their goals and 
pribrjities . If the difference between performances on the two 
simulator conditions is large, then support fot a multifactor. 
progrjamiis enh-anced and the t-ime invested in the preliminary 
effort hWs not been wasted. For example,- it will have provided 
a nieaps of trying out the equipment and the experimental . , 
personnel. It would hav..^ enabled the software, , particularly 
that dissdciated with measures of .performance , to be fully 
developed'^ and evalv^atcd. It gives a chance for the procedures 
on running the study to be smoothed. All of these would be 
done under le^s de:Mndii>ig circumstances than would be found in 
a. fuil-scalle screening stnidy , 

INITIAL OVERALL SCREENING EXPERIMENT 



ERLC 



screening experiment will be conducted to assess the 
ef fectjs of approximately 13 factors, associated with the visual 
and motion systems, the task, and the pilot, on pilot perfor- 
mance .effectiveness- in a simulated carrier-landing mission. 

Tne candidate- variables currently being cons'idered for 
inclusion in the first experiment\are: 
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a. 


Image quality (MTF) : 


Garrier 




b. 


Image quality (MTF) :' 


Seascape 




c. 


Image quality (high- 








light brightness) : 


Seascape 




d. 


' FLOLS systems 






e. 


Field of view: 


Seascape 




f , 


Velocity cues : 


Seascape x- 


~y motion 


g- 


Altitude cues: 


Seascape z 


motion 


h. 


Platfo:&^n motion 






i . 


G-seat motion 






J • 


LSO assistance 






k. 


difficulty, turbulence 




.1. 


Task difficulty, A/G 


Weight/ 




m. 


Pilot carrier-landing 


experience 





•This experiment has been discussed in some detail iri^ earlier 
g^apers , (Simon, Vreuls, et al, ,'1977 ; Naval Training Equipment. 
Genter , i;976) . A fictitious example of how it would' be 
handled isdescribed in Section V/of this report. 



VISUAL SYSTEM SCREENING EXPERIMENT 

Because of the importance of the visual system in .the ^. ^ 
AWAVS program, otaer experiments should follow the initial 
multifactor experiment ; ior example , content of ' the visual 
scene would be evaluated. . Clues obtained from the initial 
screening experiment can . indicate which visual scene variables 
that were studied^ are the most important . .it carf also 'indi- 
cate which, conditions of -the motion- isystem are .likely to affect 
design considerations of the.visual scene. But there is a 
need for a more detailed examination of the -visual scene, 
particuiarly in regard to content. The screening paradigm' 
• lends itself par^ticularly to ..such a study, namely . ability to 
study, the effect .on performance when certain objects , details , 
and informational clues are present or abseht' in the visual 
scene, as'well.as when certain physical" parameters that affect' 
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picture quality are set at less realistic levels. Given a 
large number of such variables, the screening study will permit 
them to be ordered according ;to their effect on performance in 
the carrier-landing mission. Later if considered necessary, 
f.or the quantitative .variables , a more^precise estimate of the 
function relating them to performance can be obtained with 
relatively little additional data collectioh. 
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• • ' /■■ • • SECTION ]V. 

REFINING EQON.OMICAL MULTIFACTOR ;i7feiViG'NS 

While the basic mult ifactor apprt^^^i^t^ ds .^ell understood 
and IS unquestionably "the most inforp^y^f^ and economical method 

'by which controlled -.exper imen fs"o ^SM'^^'iype can be performed, 
there is still the need tt roflpCA^it^Wnhance its applicability 

.to behavioral research . - This^j^^f •• m^s^ar'i; since it- was orig- 
inally developed .for us4 in^^ppJ^^ disciplines -- chemistry , 
agriculture, hiologf ji^^::yi;^:tmt always fit' directly the ^""i- 
peculiarities of h0ha.j}^§^^'-;xi^pctarch.': Individual techniques ' 
employed in this aj^^jpM'r^.:'ko''h^ one aspect or the ' other 

of the expertm^»1^:^?|^ss may, in some^cases,. be combined to ' ^ 
further impcp3«^.^^v^i:otal capability.- During this period of 
the cd.ntr«pJr.^^?r#'^tiWiber of techniques believed potentially 
releAmiiip^'l^^f "A"^ were investigated. (Note-' 

p4^^^3$ir this section requires some, backgrouiTd knowledge . 

.^^^^n^ll72.,_J^^_3-^M-7^V'^r977a, 1977b). 

'TROM RESOLUTION IV T9 V DESIGNS ECONOMICALLY ' ' . 

Screening designs are fractional factorials, generally of 
Resolution IV. This classification means that enough data will 
be collected.tq permit all main effects to be. isolated from 
one another. and. from ill two factor interaction effects • 
However, the two factors interaction effects are not all 
isolated from. one another; instead they are .aliased :in groups 
of independent^ strings-. ■ ■ . 

Once the critical factors, have been identified in ^ the " 
screening study, . the investigator may 'wish to derive an equation 
in the form of a polynomial that approximates the response 
surface of. proper degree.. He will not want to start a new 
experiment; instead the economical approach .would be t6 supple- 
itient the data from the screening, study until at least a 
second order or higher order surface can be approximated.. The' 
classical central composites design . is one popular data 
collection pattern for approximating response surfaces. The 
primary structure for this design is the fractional factorial. 
Resolution. V. A" design . of\ that resolution is capable of ' 
isolating all main and all two . factor interactions" frpiti one . 
an^ che.r. Thus , there is a gap between the size of the fractional 
^5^ . 2^^^"^'°!^ ^'^''^^"^"9 design at the end^ of the factor 

iderit if icatron phase, and that of the fractional, factbria'l at the 
beginning of the. response surface phase-. The question is: Whit 
IS the .most economical method of collecting the data required 
to fill this gap? . . 
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There were a number of papers in the statistical literature 
that had appeared potentially useful for solving this problem. 
The following represent some of. the papers that were reviewed: 



Draper/ N. R. and Mitchell, T. J*, Construction ^; 
of the set of 256~run designs of resolution « 
5 and the set of even .512-run designs of 
resolution ^ 6 with special reference to the 
* unique -saturated designs . . The Annals cOf 
/ " Mathematical Statistics , 1968^ 39, 246-255* 

John, P. W. M, , Augmenting 2 , designs, ' . . . 
Technometrics , 1966, .8, 469-480. 

. Pajak, T. F.. and A'ddelman, S., Minimum full . ^ 
sequences of 2^"^ resolution III plans/ 
J. Royal Stat. Soc, Series B ,. 197.5, . 37 , ^ 
88-9 5. 

.Whitwell, J^.. C. and Morbey, G.. K. , Reduced • 

. designs of resolution five , > Technometrics , 
V '.^ 19^1, ,3, 459-477. . . . " : ' ! 

Addelman/ S., Symmetrical and asyminetrical 

fractional factorial plans. . Techhometrics , 
' \ ~' 1962, 4/ 47-57; * . : ' 

• v Addelman-, S.', Sequences of two- level- fractional 

factorial plans. Technometrics ,, 196-9, IL, 

- .'477-509 ..- , . . ' ^ , 

• ■ * " ■ ■ . • ■ • ' . • -J ' ^ 

Eabh represented spme fo.rm of sequehtiaL approach to the 
Resolution V design through' a series of biocks in which moire 
sources of variance were isolated as more blocks of ^ data were 
collected. The economy of this approach lay in the fact that 
the investigator could stop the data collection when all crit- 
ical 'Sources of variance had been identified. 

Aft^r examining these and other: papers , it was decided 
they, of fered no solution for the immediate problem since the 
initial blocks, were not always the same ^3 those used in the 
scrieening design's to be used in AWAVS , and when preplanned 
blocks are used more knowledge i^ assumed than is ordinai^ily 
availabliB . They may Result in unnecessary data ^collection: 
While other uses might be f bund for these techniques , it was 
decided that for the AWAVS j3roblem\ individtlal i-<3Dlation of. 
: critical sources still seemed to be the best apprdach. This . 
means; that for any string of. two factor interactions showing 
a critical overall effect/- data would*- be- collected to isolate 
which« interactions accounted for ^the effect/(Simon, 1973, pp 
:'116-125; Daniel., 1962; 1976) . Since the primary purpose in 
,AWAVS is identification .rather than^r^sponse surface at^ 
.least initially this procedure seems the most straightfor- 
ward, and least expensive. The same would hold; true if there 



NAVTRAEQU I PC 7 7- 6- 0065-1 



•IS evidence that some three factor interactions might be present 
. and biasing, the main effects with which they are "aliased". The 
individual, rationally guided search seems much more effec- 
tive and' economical than gross procedures for collecting blocks . 
.of data mechanically. .\ 

If the critical t\^?o ^factor interactions are isolated from 
the others, even though all two factor ' interactions have not 
• been isolated, the result is for' all practical purposes the 
same as..if a complete Resolution .V design had been used. This 
.is referred to as .a "reduced . design of Resolution V." 

SCREENING DESIGNS WITH SOME 'FACTORS AT MORE THAN TWO LEVELS 

•• When screening designs involve qualitative f ac-tors , the 
investigator may wish to include more than two conditions of ■ 
a- particular factor.. For example, in AWAVS there might have 
been three, or even more distinct techniques -for superimposing 
the ship scene on the. background scene". Had this been the case,' 
therevouldbe no good basis for selecting which two shduld-be 
used for the extreme cases needed' in. the screening designs 
Occasionally, even with quantitative .factors , a design for 
handling, a three level factor might be. needed.- There are . 
times, for example, when a factor is not' for all practical 
■purposes continuous, and an investigator mighi: wish to treat 
it as- qualitative. More important' are those factors that may 
show a totai reversal in -performance level over it:--, range, 
sometimes referred to as a U-shaped performance curve. In that 
case, ian investigator- might wish to include ."a thir-d level' ■ ■ 
during , the screening process rather ' than try to guess where, 
the bend occurs in order to set one, of the two- levels at -that - 
point of. maximum effect. How then might a three or four level 
factor, be included in the conventional .2'^ "P fractional factor- 
ial used -as a'' screening design? , ■ 

One might make the^ three level factor comple'tely orthog- 
onal to the other factors in the screening design. That would 
mean that the fracti9nal factorial would be repeated three 
times, once each combined with a different ^ levels While 
this is. a clean approach, it might prove to be uneconomical. 
It . would be more so if there were i;our conditions i^n the ' . . 
qualitative'- factor i-, . . - .. . 

- '. . . ■ ■ ■' ■ . .' '.'■.' ' ■ .. 

There a'lready exist mixed level 2 "^3^ and 2^4^ fractional ' 
designs th^t have 'been published, however-, ■ these are usually 
limited t6 Resolution V fractional factorials which would be 
too 'costly to use" for screening purposes . ■ 

Stiir a third technique is to modify 'the screening desiair 
to include a ; three (or .four) level factor'. This can' be done" 
economically by .applying the .Principle, of Proportional . ' 

Frequencies.- to the- 2^^-? desigh. This principle states that 
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. a n'ecessary and sufficient condition that the main' 
effects estimates of two fa^ctors will be uncorr:elated Is ■ 
that the levels of one factor occur with each of. the levels 
of/^^ other factor with proportional frequencies . " Further-" 
rn^psrr-it also states that . . for main .effects to be • • 
.o^^ogonal to* two factor interaction effects ^ each combination^' 
of the levels of two factors must- occur with the ..levels of' 
another main effect with proportional frequencies ' 

Employing this principle , Addelman (1963, p, 60) shows 
..how three two-level factors can be replaced by one. four-level * 
factor. Then he shows how a »four level factors-can be callapsed 
• to .form a three Tevel factor, employing the same principle. 
Neither method is difficult to understand nor to do and so the' 
details will not be repeated here. Because three of the , two-' . 
level factors in the scree^ning design must be sacrificed to \ 
Include a three or a four level factor in the new design^ the 
number pf factors that' can be screened in this modified design . 
is reduced. There are times , therefore, when. . the size of the . 
screening design would have to be. increased to handle the 
desired number of factors, ' . ' ' . 

■'a • ' ' . _ _ • ' ' p ' ' ■ \ 

If trend robust screening designs are used,, the three or 
four level factors 'will not be as robust to . trends as the 
.-.individual original factors. Some . combinatiorts , however , are ' * 
better than others and must be ^discovered for each design, ^ 
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SECTION V 



f ■■ APPLYIN(5 . ECONOMICAL MULTIFACTOR DESIGNS TO • " 

AWAVS PERFORMANCE EXPERIMENT -- AN EXAMPLE 

In this section a fictitious example will be .used to show 
liOw economical -multif actor designs^ might be applied to 'an AWAVS 
performance- experiment. ; ^ ^ . . 
• ■ ■ ■'. ~ ^- ' ' ' . 

_ :^ To resell this phase of the research program, it is assumed 
■'that the equipment has been built and debugged, both experi- 
. m^nters and pilot subjects have been properly and adequately 
briefed, the list of candidate factors has been chosen by. 
e^xperts after an informed analysis, appropriate performance 
irteasures have been selecteci>, and the hardware and software 
required to obtain" and analyze the information/ either on-line 
or shortly thereafter,- have been checked out. It is also 
assumed that* this behaviorar study is a dedicated one, that is," 
allwho are. involved .with it have set as a primary goal the 
collection of' inf oripatiou' that will "be of practical' and 
enduring v.alue. " 

■ ' ■ > . ' ' ■ . . . ■ , 

IEXPERIMEN.TAL OBJECTIVES ^ 

' The experiment will have two primary objectives : one, to 
determine which of a'^large list of candidate factors supplied 
by experts have non-trivial effects on pilot performance for 
the specific task in' the simulator ; two, to obtain a response 
surface that describes the relatiohship between pilot perfor- 
mance and £he simulator parameters for the specif ic task , . 

.EXPERIMENTAL FACTORS- . . 

' , A list of candidate factors prbposed'f or the first major 
AWAVS experiment on daytime caxrier landing include the 
'following: nine Simula tor factor s , three task difficulty 
(environment) factors,, and one pilot experience factor. These 
are. listed on page 29 ^of. this report. 

. Each factor will be studied initially at two levels, or . 
two^vconditions . The levels would ''be set at practi cal limits ' .. 
of the operational space. The .two co,nda4:ions might be ecich of " 
.two alternatives, selected to represent the '..maximum of 
dif/iculty, or they might be the^ presence and' absence <of some" , . 
simulator characteristic. Subsequently other levels could be. . 
Added if they , exist and if the. addition is. warranted from an 
interpretation of the data already, collected. 

EXPERIMENTAL SUBJECTS' ' . . , \ \ - 

Pilots already 'capable of flying tHe simulator with minimum 
training would be employed in the .first experiment. This is a 
..performance study, - not a transfer of/training study. Two groups 
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with distinct skill/experience levels would be used. One would 
have practically no cagtrrier landing experience; the other ^wou Id 
have had considerable carrier landing experience. 

EXPERIMENTAL PLAN AND PROCEDURE 

The firs^t step of the program is to identify which are the . 

;critical factors in^the long candidate lis.t. The strategy here 
is to avoid wasting ^time and effort collecting data about 
factors that have incidental or no effect on the particular 
task. iPactors are included in* the ■candidate list because they 

•are believed to play. a role in the general problem class; but 
only the experiment can determine to what extent each plays a. 
role for the vspecific task under investigation. By quickly 
and inexpensively eliminating - the' factors of little practical 
importance, we. can get on with the business of understanding 
the effects of the critical factors. . . - ' 

The identification process dan best be achieved through 
the use of a "screening" design (Simon, 1975^ 1977a, 19-77b) . . . 
There are several types of screening plans that might be .selec- 
ted deper.ding on the availability of sub ject^ . anSl whether we.. 
intend to test each subject on. all experimental conditions "or 
^nbt. -It is impossible to discuss here . all of the alternatives 
that must be considered by the experimenter and 'the nuances 
involved in selecting one. or the other. Therer is. no cookbook 
approach; the experimenter must be knowledgeable about what to 
consider, the alternatives available, and the consequences of 
'each decision . We. will, by way* of illustration, select a 
particular design that would permit ^us to test 'a pilot on all , 
experimental conditions, without concern for the more common 
trend effects linear , quadratic , and cuhic — that might 
.bias 'the effects of interest. , If skilled pilots are used and 
precautions taken to minimise trial-tb-triall carry-over ^ 
effects, as an initial effort, such a s.tudy( can provide an 
iirtrnediate. overview^ of the problem and provide clues as to what 
the next step should be*. , 

The data collection plan would be a Resolution IV design 
of the form shown in Table 4 that is capable of estimating 'the 
main effects of up to 16 factors independently o£ two factor . 
interactions -by testing performance on 32 experimental condi-\ 
tiohs . The special . feature of this particular screening plan ^ 
is . that the experimental effects, e.g., of the. simulator^ and 
the task difficulty factors, will be. minimally biased if ' there. 



. An al.ternate approach would be' to_rim_a-.-d^^ 
on ,each •Bxperiment^l:.^^^ configuration)^:.- 
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TABLE' 4. - DATA FOn. 2^2-7 ■ 
' ^ TREND' RESTST/^J^T'-'; 
SCREEN IWC- DESIGNS 
N - 32,- ■-. ■ 



smaller than 1% 
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are linear> quadratic, or cubic trend effects (e,g,, subject • 

learnin(3> equiprtient -drift) running through the data. This 

particular Resolution IV screening design is said to be robust, 

or registant, to trends (Simon, 1977a). . > • . 

Pilot experience may be treated as any other factor and 
included within the experimental design, or as in this example, 
may be introduced as an additional factor, outside of and . 
orthogonal" to the design. .The decision to^include an experi- 
mental' factor within or outside the design — for example, \pask 
difficulty factors might also be added outside the screening 
design --depends on logistical considerations balanced against 
economy and information quality. . In our example, we. will keep 
the. twelve factors within the screening de,sign and pilot exper,-^ 
ience outside it. Thus, 'in this example, each pilot used will 
be tested on all 32 conditions, and at least one^high and one , 
low experience pilot would* be studied. ' ' , 



•Before continuing with, the description of the experiment, 

- ' .... _ . . . . r:- . 0.7 

IV 



let us examine the characteristics of this particular 2^^-^ 



_^creening._jjle^sj^gjl 
^^Perimental conditions purposely selected out of a possible 
2 = 4096 in. the complete factorial. Each row of the experi- 
mental design represents a different experimental condition. 
The plus pr minus sign in the column under- each If actor tmain 
effects only) shows which of the two levels the /experimenter 
would use when setting up each condition. Conditions are to 
be ruii in the order shown . ■ / 

The considerations involved in handling /ultiple perfor- 
mance measures, th^ dejpendent variables , are /much /too compli- 
_catM..to._dis„^ Therefore, -for this /example, we will 



assume that a decision has been made and fot each condition a 
single or composite performance • scdre has t/een obtained. The 
.experimental conditions are selected so that we base our ^ 
estimate of the mean of each condition of /each factor on 32 
observations. We can estimate "the main effect of each factor 
independently of one another and of any two factor interaction . 
Each mean, however, will be aliased witl/a string of three 
factor interactions. The effects of stjll higher-order ^ 
interactions are also aliased with thes'e effects but can be- 
ignored since the probability that the^ would have any 
practical effect Ls negligible. Sincd the design is capable 
of handling up 'to 16 factors ania we w^lll" use only twelve 
columns, the design provides some information regarding 
strings , of three factor interactions' not .aliased with main 
effects. The effects of string? of /two factpr .interactions 
with eight or fewer different interactions per string can be 
estimated independently of one another and of . the main and 
three factor interaction effects. (This data provides clues^ / 
regardir^g the presence of critical: two factor interactions. / 
In the screening phase, knowledge jof interactions is only 
important if it affects the selectiion or elimination of a 
factor. ! 
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. A /prime feature of this design is the order in which the 
experimental conditions are presented to the^ subject for 
testing. They are ordered in the de^sign so that when the 
data is collected, no main effect will be biased by ...any linear 
or quadratic trend running through the data and only two 
would be affected trivially (1 or 2' percent) by a cubic trend^ 
The Actual values are shown below, the experimental • design in 
Table 4. This is an important advantage when a single subject 
is tested serialj.y . The resistance to trend occurs' with , this 
.design without having to reduce the economy of the deisign by 
adding more conditions or counterbalancing^' the ^ones that are 
.used. * 

If changing the level of .a factor is difficult or time , 
consuming, then the proposed experimental design per se. is 
cumbersome. In the AWAVS experiment, changing the circuit 
boards for the MTF of tfcie carrier image may become very time' 
consuming since the equipment must be ' turned off during the 
change arfd then warmed up after it has been turned on again; 
delay can disrupt a subject's rapport. Several methods are 
--^ vad~l^b-ie~-t o- -h an d^^^ 
could be pulled outside'- tha -design and changed only a 'few 
times while the remaining factors are nested within it. Two,, 
the' design shown in Table 4 can be modified in a way that 
will reduce the number of changes required. In making this, 
modification, however, the degree to which the design is resis- 
tant to trends is diminished slightly (Simon, 1977a) . Three, 
the. best meithod, when feasible,, is always tp modify the" 
equipment to simplify changing conditions. While possibly 
initially costly, for any extended research program, it can be 
justified by the savings in time and the improvement in data 
qiaality^ - - - " - , " 

Analysis of the First. Set of Data 
* * *' . 

Once the performances at the 32 data points have been , 
measurdd for. a. single 'pilot, whatever his experience level, 
the data can be analyzed. This analysis is extremely simple, 
consisting of finding the mean difference between 'high (+) 
and low (-) ^conditions in each column. This can be ^expedited 
by using Yates' algorithm (Simon, 1977a). 

IThe results of such an analysis is illustrated (using 
fictitious data*). in Table 5. In this example the twelve 

The numbers were taken from an actual experiment, so they 
do reflect what can be expected from a real experiment. How-' 
ever, the context in which they appear has been modified to 
fit the example. 
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TABI^ 5. ANALYSIS 01'; FICTITIOUS AWAVS DATA FROM' DESIGN IN TABLE" 4. 

1 . 2 3 



I Rank 
/(largest 1st) 



Source 



-Mean 
Difference 
(Effect) 



Eta 
Squared 



Gumulative 
Proportion 
of Variance, 
Accounted For 
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Represents a striny of three-factor interactions 
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simulator ^and ,taBk difficulty fcK'-tors were includod in the 
design and Lhc problem of level chancjinq has been solved, with- - 
out mOGi.fyinq the design. We will examine the f^.ctitious 
results from a sinale pilot tested on all 32 conditions . 

The results in Table 5 listed the. effects of each source 
of variance — ma'in, two factor and three factor interaction 
strings in order of their magnitude (Coi. 1) The . proportion 
of the , total variance contributed by each independent source, 
is shown in Col„ 2. The cumulative proportion accounting for 
all" sources as each succeeding one is included is shown in 
Col. 3. • " ' . 

The investigator must decide which sources of variance 
are critical. Within some -reasonable limits he can probably 
state what- minimum size effect (difference) he considers to be 
of practical importance. He will ordinarily have little diffi-. ^ 
culty eliminating thpse ' very small effects that would be- 
considered trivial . He can also recognize the. obviously 
critical factors which have very large effects^ Therefore , the 

major probjem for the investigator is to decXd_e .whXc.h__cif__th^ \ \ 

marginal effects . are to be considered important. Let us say 
for this illustration that a mean difference (an effect) of less 
than .10 is probably trivial. That would mean that Factors E, 
A, and G are probably critical, while F and K are marginal for 
this particular task (and within the limits set by the experi- 
ment) and. Factor D is right on the line*. If Col. 2 is examined, 
we can see that Factor F accounts for approximately eight percent of 
the. variance in this experiment and FactorxK accounts for three 
percent. The other three (E, A, and G) areNm^rk^^ 

:If 'we examine Cbl/ -3, we 'see -i:hat :fdry m^^ Effects only, if Factors 
A, F, and D are terms in a first order polynomial'^ -the -re-. - 

gr^ssion would account for approximately 66 percent of tli'e total 
variance. - 

* -'If the effects oF all .. soutces up to and including Factor D 
were included in a regression equation, we would account for 81 
peircent of the total yarj.ance. If all sources up to and includingV 
Factor K were included in an- equation which would be ess^entially ' 
a first order polynomial with an additional term representing a 
string of three factor interactions, we would account for 76 percent 
of' the performance variance in this experiment. The 76 percent re-_ 
presents a multiple, correlation of .87, which is respectable since it is 



/ 
/ 



There are other considerations that would be involved in 
this interpretation, too detailed to describe here. Once 
again, the investigator cannot analyze his data mechanically; 
he must understand the process and apply it wisely. 
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based on five factors out of twelve originally believed impor- » 
tant by a group. of experts., and in fact, represents a prediction 
based on all 12 — for this, task, sub ject type , and within the 
limits of the experimental conditions. . 

But we cannot arbitrarily add or dismiss sources of 
.variance in this way. We could make ourselves "look good" by 
adding more and more, though it would have 'little meaning 
operationally. We ne^v. other ,Cri Leria to make our selection ' 
at the point the di fferences approach the trivial level and, the 
proportion of. variance accounted for by each-new addition is 
small. Although there were no '.replications, in the design by' ■ 
which to estimate an .error variance (this will, be further 
discussed later) , we can use ojrder statistics to -estimate wha,t 
the error variance is ,and whether .an observed effect is larger 
than one might' expect to find by- chance . ■ 

In Figure 2, a half-normal plot is shown of all 31 effects -- 
the. mean differences -- of the study. The slant line represents 
a normal distribution of aset of effects. All effects located 
to, the right of this line" would therefore be considered larger 

-fehan-ene-mighir-expect by chance. It is clear that neither the 
effects of D nor K in this study were larger than might have 

■been expected 1 chance. TJJie four factors E, A^ G, and F, 
along with the. string- of triple interactions, accounted for 72 percent 
of the variance, yielding a. multiple correlation of .85-+ .10. 

The study would be . repeated using the pilots with different ' 
amounts of experience. Examination of both sets of results, 
separately a'nd in combination, looking for patterns and for I 
marked differences, .would. be an important part of .the analysis. 

- ... . ■ 4^- 

Having reached this point, an investigator has a number 
of choices. > If the only purpose of the experiment is to 
identify the critical factors, we have come close to it already. 
Whether or not Factor ,K Or a.r}y of those with even smaller 
effects would be used at the level (configuration) producing the 
highest performance is no longer a decision based on performance. 
Since _ the differences in performance are marginal, costs and 
technical considerations become the overriding criteria. In a 
program such as AWAVS other criteria , e.g.,. transfer effec- 
tiveness,., can also determine which configuration .would be used. 

The first objective of this experiment has still not been 
met until we have ^answered a few more questions., One of them 
is: What interaction (s) within the string showing the large • 
composite effect actuallj^ accounted for that effect?. It. is 
possible that that interaction might include a factor that was 
not one ;of , the four selected as critical. In other words, before 
we can be sure we haye not" omitted a""cr it icai ~ra^^^ we sho"uld" 
collect some additional data to see which one of the triple 
interactions in the string (listed in Table 6 ) was responsible 
for the large effect. 
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Figure 2. Half Normal Plot of Experimental Data in Table 5 , 
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TABLE 6. THREE FArTOR INTERACTIONS^ IN THE 
CRITICAL STUIlsG IxN TABLE 5 



ABJ ACI AEF BCE CEF 

CGK DGJ DHI . EGH, EIJ 



If we wish to isolate ^the effec'ts of each of these inter- 
actions from one another, we would have to collect performance 
data . at a minimum of ten new coordinates, although . for the 
sake of balance, .16 would probably be used. However, we can" 
make some preliminary guessed that might reduce the effort. 
For example, if we only considered the interactions that were 
'composed of some of. the four factors that we knew were 
critical., we Would only have to isolate 



AEF , 



• This IS also the one identified if we were to consider those" 
, containing all factors in the upper half of the plot. In this 
way, if we find it does account for most of the observed effect 
in^ that string, we'd not have tp collect any more data. In 
theory, we could estimate the. effect of the- AEF interaction in 
the same way we estimated, the effect of Factor A, by finding 
two conditions, one of which represents the + c^dition of 
Interaction AEF . and one which represents the -Trcondition of AEF 
g^^jD-UBJLy,,„„conditlQns and (1). would serve these requirementi 
Also abef knd b. anpf pinr\ r« ^Kr^H^^-F ^y^a -.v.,q jr^_a_i- 



Also abef and b, acef and c, abcdef and bed, . and so forth. 
Several of these might be used to increase .the reliability of 
the estimate. 

If the magnitude of the AEF' effect did not correspond 
-wa-th-tha-t-founa in the study- ^- and one must allow some lei^ay""" 
for differences m. the. data collection process — then one must 
look further and begin to suspect that . the critical interaction 
IS a disordanal one. In this example, however, it would be 
highly unlikely that this were the. case, .but-: i.f it were 
necessary to isolate the remaining sources, a balanced plan 
(see Simon, 1973, pp 120-123) might be employed. 

The chances are good that this, quick approach will work 
since most interactilons found in the behavioral sciences are of 
the ordinal type, •■-.n that case, the large interactions would 
be associated with the large effects and can be eliminate'd by ' 
>res::aling the dependent variable. The less frequent disordinal 
typo of interaction is the more important one, with which the 
interaction may be large while the main effects making up 
those interactions might appear trivial. Since these interac- 
txons cannot b e-eliminated by some transformation of the~dart^— 
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they are sometimes referred to as' "intrinsic" interactions. If 
we wish to be certain that we have found all% critical main 
effects, we must be certain we have detected any that contrib- 
ute to disordinal interactions. 

* , ■ .5 * 

In this example, no strings of two factor interactions 
were found to have critical effects, although the one set 
(located in rank between 'Factors K and D) might be a possible 
candidate. Ordinarily, there ±s a greater chance of having a 

- critical- two factor effect than a three factor effect. It is. 
interestiag to note that although this string did not show a 
large effect,, interaction AF was in the string; With Factors 
A and F. and interaction AEF all large., it is no£ surprising that 
the string with AF was also large; ^hdwever, inspection of the 
half-normal plot (Figure 2), suggests" that an effect of^^this 
magnitude would probably have occurred by chance. Whether in 
fact it did account. for. the proportion of variance^^shp^^n . in 

. the string would have to be tested by the additiotT^oT new 
experimental conditions as was done in the case of the three 
factor interaction. 

— ~, — '■ — There "is One point~'^ha'r~should be , remembered in regard to 
strings of interactions: it is possible for two large effects 
to cancel one anotner. While the chances are not necessarily 
high, the inveistigator must be alert for that possibility. The. 
analysis; that should precede an experimental effort will often 
supply the investigator, with the cues^ necessary to anticipate 
this situation. . ^ 

At this point in the investigation ^ we .should have identi- 
fied all of the critical factors out of the'' candidate group, v 
including thosp that might have 'l)een hidden within a di'sordinaJL 

-inTteirairrtTUir:^ — Th'e~co^t~o"f~:sra'Ch"~~an^ e to sttdy 12 equipment 

and envirdnmertt factors plus pilot experience in the manner 
proposed, would be the costs of collecting data on. 2 X!.32 = 64 ' 
observations, plus possibly an additional twenty or so 

* observations * Had we decided to. make the subject 
factor ■ a p art of the Resolution IV design, then the study 

"TrLrglrETiave Been colicludM with a f ew as 50 observations . 

. Certainly this is sufficient to obtain the information of any 
practical importan6e. 

About the only weakness, at this point is in the assumption 
^that two experience levels are sufficient to classify the pilot 
^and that all pilots within these- two groups .would in fact be 
homogeneous. If they are, th^n our experiment , insofar as 
objective one is concerned, is complete. If they are. not, it 
is not the design that is at fault, but the original planning, 
for. the intent- is^to identify all critical factors including 
pilot characteristics'^ that might /influence .simulator design. 
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More*' pilots may have to be run i\\ the latter case, but hot in 
a haphazard manner. Identification of the other pilot dimen- 

-sions becomes a crucial issue, somewhat oblique to » the 
original objective but one which could influence the interpreta- 

. tion of the results. In practice, it is highly unlikely that 
only one pilot of each type -would ' have been run; still it is 
important that when more are included, it is because we wish to 

- extract more information, not that we just wish to be redundant. 

O btaining a Regpons.e Surface ^ \ . - 

A response surface is merely a representation of the 
multidimensional functions relating performance to the critical 
experimental.' factors .. It is frequently represented by a 
polynomial equation derived from: the experimental data. While 
an equation can be written whether the factors are qualitative " 
or quantitative^' continuo.us ^ or discrete, the concept- of a 
respgnse surface implies that the variables involved can be 
described along a: continuum. 

; In ■ the pr Imary AWAVS s tndyv"a^--xtr'ha s--been plann^ed^.L^rnQ'^t 

of the factors are either Qualitative or dichotomous and jj^ 
discrete quantitative factors, and as such, do not need to be 
represented by a response surface. For all practical . purposes , 
the experiment would stop when all the critical factors Had 
been identified and the best configuration identified. However, 
for purposes of illustration, we shall continue this section " 
using the AWAV example to illustrate the steps involved if we 
wished\to approximate the best fit of- a response surface were 
..the • variables of the appropriate type. ' 

The data from the screening design can be ^usjSj3_to_wr±te, 

an equation containing only linear terms: 

Y . 543 + . 168 E + .121 A + .113 G + ,0.90- F - .113 AEF ^ 

with each coefficient equal to one-h^lf the mean difference for 
. th.e„-_CDXnes];iondiag„ ef f ec^b^ The -in ter ac tion AEF i s* a 1 inear 
interaction, i.e.,. linear A. x linear. E x linearF- Before final 
acceptance, the xesiduals from this equation should be analyzed 
(Daniel, 1976).. 

• If an investigator plans to develop a response surface, 
he should include center points in his .experimental design 
..during the screening phase. These center points are. at co- 
ordinates (0, 0, 0, ...0, 0) in the center of the incoinplete 
hypercube,. defined by- the 212-7 fractional factorial,'. Several 
measures at the center. wouT&, be taken, preferably at equal ^ 
intervals alonq. the 32 condition run. Since we will continue 
this example and assume that the 12 factors . were in fact 
quantitative and continuous, we will have already . included 
center point measurements - of performance at the beginning -and 
end pf the 32 condition run and after the 8th, 16th, and 24th. 
conditions, making a t ota l of fiv e center pnintc; .in ^11, . 
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At this point," we do not know whether the linear equation 
shown above adequately represents the true response surface. 
It is not uncommon to find the relationship between performance 
and factor's in behavioral studies to be non-linear. The 
center point data provides us. with sa opportunity to test to 
see whether there ia curvilinearity in the response" surface, 
for if all ' dimensions were collapsed onto a single dimensionj,. 
we -would ha've measures at three levels of each factor, enough 
.to test to see if ^ quadratic relationship would better describe 
the data\ If a" Lack of Fit Test reveals that the linear 
equation is no t adequate , then the investigatpr must be .prepared 
to collect more data. . • 

His first gpal is. to collect enough data to write a second 
degree polynomial, which would include all critical main 
' effects ,; all critical two factor interactions, and all critical 
quadratic terms. In this study, v;e have already determined « 
that ~^riTrear two factor interactions have probably only trivial . 
effects and^ that there is one important linear triple inter- 
action and so in that regard, we, are ahead of the game\ Still 

we w ill want to add some points to estimate the quadrati c ^ 

terijis.. One data collection plan for this purpose is called a 
"central-^composite" design (Simon, 1970b, 1973, 1976a, 1977a, b) , 

• k-p 

The classic, central-composite design is composed o;E a 2 ^ 

'Resolution V factorial hypercube, a 2k star portion, and some 
center points, where k equals the number of factors and p is ^ 
the fraction , of the complete factorial needed to satisfy the 

. requirements of a Resolution V design. With that design all 
main effects and al] ; two factor interactions would be isolated 
from' one an'other. The screening- design /'alX^^^^^^^ completed, 

provided us wi th a 2^^"'^ Res o l utio n . IV design in whic h all 

main • effects were/ estimated independently of one another. and of- 
the two., factor interactions, but within sets of independent 
strings, two factor interactions were still aliased with one 
another. Ordinarily the investigator might collect more data 
to make the Resolution IV design a Resolution V design, or he 
may find another solution that does not . rqquire^ m 
Ther^ is such ^a solution in this example,. 

From the results of the screening Study, it had been 
Concluded that only four factors were critical . If we were to 
drop all^ letters. 'representing the non-cri*tical f actors • from a 
completed design in which al^l aliased two factor interactions 

"^are shown ,. i .e . , Table V, we would find that the original 
2I.2-7 Resolution IV design becomes, for all practical purposes, 
a Resolution V+ desigp. ' Had the three factor interactions in 
the strings been listed, it would have been seen that effects 
of a complete 2^ factorial are estimated since all othQr^ 
effects were judged trivial. Note that the six possible, two 
factor interaction terms for the four critical factors are al 

. estimated' independently of one another at ranks 25 (AF) , '// 
(EG), 17(AE)', ll(Fr.), 7 (EF) , and 1 (AG). The effects of th^se 
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in this 



experiment were judged -to be -inconsequential . Thgre- 
foro, althoucjh DO more data has boon' collec ted ,. we have, for 
■all practical ■ purposes , the Resolution V design required for 
the fractional hypercube portion of the central-composite 
design . " 



• In fact 
the investig 
albeit margi 
mate the ten 
all independ 
interactions 
and" 4 (EK) . 
15, 14 ,13, 
make, up the 



if as a precaution in writing the response surface 
ator preferred to include Factor- K in the equation, 
nal, the existing data is still sufficient to esti- 

two- factor, interactions for these five factors, 
ent of one another. .The additional two factor 
can be found at ranks TZ (AK ) , 21 (fk) , 20 (GK ) , 
The remaining variances at ranks 24, 23 , 19, -Ig , 
12,' 10, 9, 3/ 6, 5-, 3, and 2 would be combined to 
"er^ror" variance*. 



If; we perform a Lack of Fit test — using the center points 
for this purpose — and find that a . test ofthe linear fit is 
■poor, then data should be collected at the "star" points to 
estimate the coefficients of the quadratic terms for the five 
factors^ .These. -poin j bs are l Qmt4d_a:t-_CQOxdxna-te-s— (^-a-,--0-rO^-rOT-)- 

(0, +a , 0,0,0,), (0,0,0,0, +a) . The value of~a depends on 

other features of the design, and a discussion of how it is 
selected is -too involved for this paper. The central-composite 
design- .requires that the number of star points "equal two times 
the number of factors in the experiment. Therefore, if • the 
investigator, decides to keep the five factors, he must collect • 
\data at a minimum of 2 x 5 = 10 additional points. When the 

ir points are combined with thfe points of the fractional 
hi^ercube^ and the Center points in the screening design,, five 
me^urements will have been made along the scale for each 
_J^5-.^g?.:^_.3h.ile t^^^^^ design, the 

points are located . so that estimates of the quadratic terms • 
can be\ obtained. ■ ■ ; . " ; 

■ .Sihce we presumably had identified all critical two and 
three' factor interactions during the screening phase by col- 
lecting da\ta. at.a, to.tal of^.3 plus 5 (center) plus^O 

(star) equaXLs 47 experimental conditions, we have approximated 
the responseX surface for a five factor space. However; it 
should be rerrtembexed that we began with a 12 .factor space of 
which only theXfive had critical effects in. the^particular task'. 
If the 12 factdrs originally selected by the experts were 'in ' 
fact the most li>kely candidates influencing performance on the 
task under investigation, then this laboratory-derived equation 
of the ..only truly \^ri tidal five out of 12 factors should be 



Actually thet^e cdntained the higher-interaction termg 
required to complete th\ 2^ factorial all shown to be 
negligible. 
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expected to predict performance under operational conditions 
qui.te well. yA different response surface v;ould "Be derived in 
the same way for each pilot experience level investigated." 

VERIFJCCATIQN AND FIDUCIAL LIMITS . ' \ , 

•.Once an adequate equation has been derived, d.epending on 
the time .and resources available , the .investigator may wish to 
do two things: 1) to establish confidence limits, and 2) to 
vetify the equation. The first might be done .by replicating*' 
the existing design at select points — -a partial replication. 
The second, might be done by selecting combina:tions of factors 
where no. previous data had been taken to see if the equation 
would predict the results within acceptable confidence limits; 
TKe realj test for verifying the; equation would be to collect 
data under field conditions to determine how clo-sely t^he equa- 
tion would predict it. Unmentloned in the aHove discussion , ' 
but critical in any holistic approach to a problem/ is' the 
handling ^of uncontrollable variables. If they ckn't be 
ntanipulated , then- they should .be..JBe^asAire.d.„and^th'eir.__eif.ects.- . 
isolated from the other data through some covariance analysis. 
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SECTION VI 
QUASI-TRANSFER EXPERIMENTS 



There has actually been very little research seeking funda- 
mental principle's of transfer* for the pilot training' situation ./ " 
Many studies have been conducted for the purpose of evaluating 
existing devices and -as such do not provide .the inft>rmation 
, . needed to optimize design. 'J Some studies performed with, the 
. intended pi^rpose of answering fundamental questions hav^ been 
so narrow in their context that it would be ..foolhardy to. 
generalise,. beyond the conditions of the particular experiment. 
^ . ^Extrapolations £rt)m the. results of classical transfer of 

training studies • — . often on verbal ijj^terial or oversimplified 
perceptual-motor, tasks — • cannot be made with confidence, at ^ 
.least insofar, as recommeoidations regarding specific design 
decisions are concerned . It is therefore desirable to pursue ^ 
studies in the context of pilot training .simulators such as 
, .AWAVS that seek principles of transfer of training. For this 
• purpose , quasi --transfer experiments can be considered as an. ' 
economical but effective approach to use. 

A /'quasi-transfer'^ experiment for the AWAVS program, is 
. defined as one in which performance is never measured und^r 
.^.realistic, i.e.. non-simulation, conditions. For ^pilot training 
this means that the' experiment, would include no post-training 
p.eribds in which' performaiice would be measured in the aircxsrft. 
Instead,, an alternate simulation configuration would be used 
- to represent the flight conditions; 

This . artificiality makes it necessary to interpret -experi-- - 
men-feal; results with caution. They may be used' to' uriderst 

^~the--1^a-ns-f-er-'^#--tiJ^a^^^ „ 

- without considerable experience and support data for • 
■evaluating the transfer of training qualities of the AWAVS 
*sim.ulator . , •Whatever\dif ferences exist between the simulator v 
' configuration, representing the aircraft and the actual aircraft • 
V differences that. may not be evident^, to the investigator 

X' from ^ 

thjs simulation experiment to the specific air>craft. - These 
considerations, howeviar, should not cjiscdurage use of a simulator 
' to understand conditions affecting transfer of , training in 
■ general. - In essenpe, we would use the quasi-'^trknsfer experiment 
to discover what transfer of a. particular natur^, quantity , ^nd . 
direction (i.e., positive or negative) would'be Effected -by 
specific simulator characteristics. Understanding these things 
in depth would facilitate our ability to make better design " 
decisions in future simulation, efforts., and help- ius to plan' arl 
-conduct real transfer studies. * ~, - . i 



FIDELITY 
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No single unprov^n principle dominates the design of / 
pilot traininig simulators more than the "fidelity principle 
This principle, implies \ that : 

Transfer ''of training from simulator -to . V 
■ ' \ ■ aircx-aft is' a positive function of the . 

degree .to which^ the simulator faithfully . - ■ 
reflects the characteristics of the 
aircraft. .. 



In' Figures 3A through 3C, ^ra:phic representations of. this 
principle along with cost c^nsideratipns have been reproduced 
from several reports on thi^ topic (Kron, 197p 1^ Roscoe,. 1975) . 
That "fidelity" has never been adequately defined -tias not ' 9. " 
deterred the use of this principle which has its roots in ' ■ •■ 
classical .psycliology studies .of tra^isfer.' For some fidelity ■ .a 

..implies physical realism; for others it suggests' that p^ychol- I 
ogical similarity is probably more importarlt. On the other 
hand, some such as 'Caro' (1973) belie</e that how the simulator 

"Ts~"u^e3rTs"lTio^r^ 

deqree of simulation realism. * " \ ' \ - 

e . 1/ • • • . ■■■ . : 

Evidence that realism is important is attributed from 
applications of simulator training, as employed by the comin^cial 
airlines to train and upgrade, pilot skills. There have ali^ 
beeA'^ cojfnponent studies (of ten unde^p simplified, conditions) / that 
.purport, to demonstraite the validity of the .•principle. Other./ 
cqi[nppne.nt..'is.tudie^ to dexionstrate that-the principle does 

not hold. Ho\yeuer, valid the fidelity principle may be, costs ' 
and Btate-of-the-art of simulation piace considerable pressure* 
7on"tfiose5" 

faithfiii reproduotion fof reality as is compatible*, with efffective 
.tr'aininq. In sjprite of large outlays, \of money for research, no ■ ) 
e.xperiment to (jJate- has provided definitive answers, nor has been'. . 
•sufficient to/specify ttlose CQnditions under which fidelity^ is ' 
required nor^'to dimensicnalize fidelity into its composite ° 
parts and. demonstrate, the conditions «under; whiclj each component' 
is importar^i: to transfeY. " ; " . , - 

Pimerisionalizing' the Siifuation • ' . • 

■ ■ :/■ ■ . • ^ ' ■ ■ ^ ' ■ • ' , 

Before the fidelity pi^pblem can be attacked properly,^ the.^ 

situatioii in which fidelity is to be examined must' be .more ^ 

thorptighly dimensibnalized than it has been in the .past. . While . 

' most of /these characteristics have been' recognized in dis- 
cussions of fidelity, few . investigators seem to .-see the need to' 

■.specify the part of -.the multidi^mensionat space v;hich their ' 
experimen,t is intended to illuminate. . Human behavior ig 
situation specific. To discuss "fidelity^'? we must dis'cuss 'it 
in the cohte>Ct- of 'a situation. Dimensions of an AWAVS sit.uation 
include : * " ' ^> * ' ' 
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NORMALIZED PERFORMANCE TOLEP 




•FIDELITY OF SIMULATION / 

ft ... " ' ■ 



Figure 3. Theoretical /Relationships Between Transfei"/ 
'-■ „ . Simulation ^fidelity and Costs 

■ •■■.'"/"• * 
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Pilot., skill 

Pilot experience 
Task complexity^ 
Simulator mission 

Simulator co^mplexity (i.e., aircraft simulated) 
Simulator component (e.g., visual system) 

Other critical considerations include: 

Training curriculae ^ 
Instructor/ skill 
Performance criteria 



Dimensional izing Fidelity 



-Simulation (ri^^dolity has generally been evaluated in terms of 

known similarities between physical systems or on' the basis of pilot 
judqment. There have been proposals wherein performance equiva- 
lence oii' a simulator and aircraft would be interpreted as 
.perceptual equivalence, implying a measure of effective fidelity. 
But these approaches have two weaknesses: 1) they presume that 
fidelity is a single entity/>and * simulator fidelity becomes a 
gross measure; 2) they don'^ answer whether or no^. faithful 
simulation is a necessary feature at all. Certainly there are 
recognized examples where a simplification in some case "or 

increased 'difficulties i-n-other-s-ha-ve— been— successfully employed- ~ — 

to improve \ transfer of training. This implies that research in 
fidelity should break fidelity down into meaningful parts and ' 
to ask the mork general^ question: Under what condi.ti'ons are 
the components; of fidelity important and under what conditions 
are they not\ in the training context? 

home exa /of the more obviously different ways in which 

fidelity of the visual or motion simulation system can\be 
dimensionalized ar/e shown in Table 7, 



Experiment s 
» 

Given; an 
relevant ,to an 



Appropriate simulator*/ experimental questiorij 
linders tanding of r,fidelity aAd its effect on 



transfer of training can ' be examined. 



What an 



/ 



\ 



appropriate simulator is will not be define 



■The answer is < probxibly— pragmatic — it will-.J>e appxapriat 



is • available ,/ 
and. repie$en|b 



has the degree of flexibility suitable for re 
3 the AWAVS type tasks. 
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TABLE 7. EXAMPLES OF DIMENSIONS OF FIDELITY 

IN VISUAL AND MOTION SIMULATION SYSTEMS 



Type of Fidelity 
Visual system: 

Continuous variables^ 
that may be decreased 
, or increased 

Spatial distortion 



Temporal distortion 

Incompleteness 

emission of objects 

N Omission of detail 

Skeletal/ pictorial 
ox symbolic 



• Added information 
Motion system: 

Simplified model 
Distorted feel. 



Examples 



Resolution; brightness., 
contrast 



Size , shape , patterns 

Speed of response; lag 

relative to compatible 
motion .system 

Realism of . background content 

Sea texture 



Attention getters; emphasizers 
not found 'in real world 



Aircraft dynamics; omitted 

\ degrees of motion 
Airei;;aft dynamics; motion 
K^naesthetic cues 
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How do reductions in fidelity affect system transfer? A . 
quasi-transfer study might be conducted using one simulator 
conf igur^ition to represent the real world, i.e., the aircraft, 
and all other configurations to represent varying degrees of 
reduced fidelity^ . If preliminary studies relating fidelity to. 
performance were conducted first , an investigator might use 
that information in planning this study. Ordinarily the most 
sophisticated simulator . configuration-might be us^d to repre- 
sent the aircraft: on the other hand , for crertain classes " of — 
variables, no particular configuration need be singled out. 
Instead the study would be conducted to find out what happens 
to transfer when fidelity increases or decreases, when transfer 
is positiye or negative as a function of the psychophysical 
characteristics of simulator components- / - 

Experimental designs described earlier for economically 
4Darforming large -multif actor per formance "•studies might be 
employed in these experiments- Subjects would be trained, on 
simulator configurations differing in their fidelity and sub- 
sequently tested for transfer on -ariother^jrmA34a^t^r— ee n f igu r^-t^-on 
representing the aircraft- In addition to providing a compre- 
hensive picture of the transfer problem in complex simulation 
and task situations^ thes^ studies would also provide a chance 
to experience, evaluate, and learn more about proposed econom- 
ical transfer designs prior to their. use under "real" ' conditions 
Some experimental data collection plans, described later in 
Section VII, could be examined in a quasi-tr.ansf er study in 
order to improve our transfer of training research methodology - 



Novel Transfer of Training Designs 

Simon (1974) reviewed a class of experimental designs, 
called "ehariqe-over , " "cross-over," "carry-ovei; , " or "residual" 
designs, that might make the study of transfer principles more 
economical if they were einployed. Unlike the designs used in a 
conventional transfer experiment, these permit a single subject 
to be tested on a number of configurations serially , while, 
beinq able to measure the residual effect carried over from one 
configuration to ^ the one that follows it. The designs are 
capable of isolating the direct, effect of "the configuration 
being tested on the particular trial from the residual effect 
carried over transferred • — from practice on a different 
configuration used dn the previous trial . 

This class -of; design lends itself particularly to quasi- 
transfer experiments, where the simulator can.be used for all 
the configurations under investigation- Each configuration wil 
be preceded and -followed by every other configuration, so that 
at the' end af- the experiment, we can / determine which configura- 
tion-has the largest average residual (transfer) effect on the 
performance of the configuration that followed it.. .If there 
are interactions between direct and residual effects so ^ that 
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the. amount of transfer due to one configuration depends on 
which particular configuration follows.it, then this too can be 
analyzed, although the designs for this purpose are more compli- 
cated. By having the configurations in the series vary in more 
than one dimension, the relationships between simulator 
components, fidelity, and transfer may eventually be determined. 

Measuring sequential transfer is not completely new to 
psychologists who have included "order" in 

t^^°se cases, with only two conditions, A and B, to 



be studied, ha-lf nDrf-the sub^ets-a^e-^n-on order A to B, and 
half on B to A, and the effects evaluated. Ordinarily this has 
been done for cleansing rather than for informative reasons. 

■ i 

Change-over designs appear in two basic forms': 
one requiring a number of subjects (where direct and residual 
effects are balanced across subjects) and the other in whichl ' 
estimates of residual effects are baJ^anced within the responses 
made by a single subject tested serially. 



For example, here is a design in which four experimental 
configurations that. differ in their similarity to one another 
along a known dimension (or dimensions) might be used to 
determine the amount of transfer that can be attributed to 
conditions A, B, C, and D: 









Subjects 












I 


II 


III 


IV 






1 


A 


B 


c 


D 


Trial 




2 


B4-a 


D+b 


A+c 


C+d 


(Period) 




3 


C+b 


lA+d 


D+a 


B+c 




▼ 4 


D+c 


.C+a 


B+d 


A+b 



The capital letters indicate which experimental condition 
(A,B,C, or D) is being tested. It's effect is referred to as 
the "direct" effect. But performance in these serial 
presentations may also.be affected \by "residual" effects 
carried over from the previous . configuration-, as indicated by ' 
the small, letters (a,b,c, and d) . Performance as it is ' 
measured on any trial is the composite of both the direct and 
residual effeqt...The direct effects are distributed - in the" 
arrangement of a balanced Latin square, with each condition 
•.preceding and following every, other condition (vertically) 
o-ice >and only once, and also . appearing once in each column 
and each row. Direct and residual- effects can b6 independently 
estimated by. adding a fifth trial .(row) in which ' the condi- 
tions of the -fourth row are repeated: . ' 

" -5 D+c C+a B+d A+b 
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The. total variance of thxs extra period design can be pa rti - • 

tioned as follows: " ^" ~ 

Subjects ' 
• ' Trials (periods) " ■• 

Direct effects \ 

Residual effects 

Eifror 

There are several variations on this plan (see Simon; 1974). 
Its major limitation is.that.it assumes that the residual for 
any configuration (or condition) is constant irrespective of 
what configuration follows it. For the most part, these 
designs are not used factoriallyy that is, the four conditions 
ordinarily do not represent a 2^ set of conditions , although 
there is no reason why they cannot. ' ' 

Other d'esigns are available when direct and residual 
effects are assumed to interact. However, these designs have/ 
Tiever~really beerT" opt inrizea~ have serdorn been usedV "a^ " ~~i 
ordinarily increase. the amount of data collection required. / 
If we seriously wish to/develop new economical methods of study- 
ing transfer, this class ol: design should not be overlooked// ^ 

/■ 

A different t'/pe.of design, referred to as a serially / 
balanced sequence design , can be used with a single subject^' 
tested repeatedJ.y on all experimental conditions. One example 
for four conditions is: 



> , ._ 

' '"' _ B; B C A D; D C B A;. A B D C; C A D B; ' . * 

/ B D A C; C p B A; A B C D; D A C B 

Block effects^ direct effects, residual effects, and error can 
be estimated with designs ^ of this type although their effects 
are not^ always orthogonal. Sequences are* usually balanced, 
against direct and residual interaction effects although in 
'the past. /these interactions have not been isblated . Both 
/serially baia:nced, and carry-over designs can be. adapted to 
measure not only first residual, but second residual effects 
that occur two trials after the d'irept effects were introduced*^ 

^ ^: Where fehe^ effort can be made at relatively- low costs > an 
attempt ' should be made to employ ^this class of design if for 
ho other ^reason' than -to establish its value for the experimental 
study .of transfer and simuTator fidelity. If effecti-ve, it can 
represent a less expensive means of learning, something quickly 
aboat transfer. It is apparent that these /designs lend them- 
selves to. cjnly/certain problems, particularly where t2;'aining to 
use ;theT^'^mirlatpr fes taken place .prior ro rhe experiment and 
suffices for all configurations. For designs in, which the 
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res idua l effects are assumed to oe additive to each direct 
eFfect, We would hope to find out which configuration is likely 
to result in the highest overall effect being carried over to 
the configurations that follow. The assumption is made that on 
a relative basis this would hold true were the real aircraft 
involved- On the other hand, if designs are used in which 
direct-by"-residual interaction effects can be isolated, we may 
discover more fundamental relationships about fidelity' and 
transfer. The only, way to evaluate their effectiveness is to 
^try them . I 



AWAVS AS A CRITERION DEVICE 

# . 

Although implementation is still a future consideration, 
planning might begin at this time regarding the use of AWAVS 
as" a criterion device for transfer of training research. This 
means that a particular configuration of AWAVS, rather than an 

actual aircr aft, would be us e d to ev aluate transfer in pilot 

traininq studies. This approach is differentiated from, that 
found in the "quasi-transf er " studies proposisd earlier by the 
addition of an e^oirical data collection effort to effectively 
equate a simulator configuration to , the aircraft. Only after 
an AWAVS configuration is so equated can experimental data 
with the simulator substituted for the aircraft be interpreted 
with confidence. One method of achieving this equivalence^ 
has been .proposed by Matheny. (1974), Some effort now might be 
devoted to a study to discover if such programs . have ever been 
implemented (and if so, their current status) , and whether 
.they might be improved upon, particularly in regard to simpli- 
'-f^:ea±^TT— and- economy . ~ ; 
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SECTION VII 

. ECONOMICAL DATA COLLECTION PLANS FOR TRANSFER 

OF TRAINING STUDIES FOR THE AWAVS PROGRAM 

'\ . " ' - 

.;If the multi factor approach is to be applied to transfer 
of training research ' (as opposed to performance research) then 
it is necessary, to find even more economical data collection 
plans that are suitable for this. class of problem. ^ The cost, 
of data collection is intensified in a transfer of training 
study over that found in a performance stud^ because each ex- 
perimental condition is first associated with an extensive 
training period in the simulator and later tested in flight in 
the aircraft. Sdme ways of reducing this burden are suggested^ here. 
It should be noted, however, that these ideas are still in a 
conceptual stage, requirir :j . empirical experience to test them 
and turn them into working plans, or to. ultimately discard them. . 

• ■ ■ ■ ■ \ '. 
Two basic approaches are proposed for econoraically discover- 
ing s i mu 1 a t o r c o n f i g u r a t-iiDn s- on-w h-i c h— t r a-n&f-err-e-f-fee-t-i-veiie s-s 

should be high. These are: 

. ■ \ ^ 

a;''orie in which ^ complete and thorough multif actor study 
of simulator factors- is conducted usirig pilots skilled 
enough to fly the . simulator without exrfcensive training . '; 
This would be followed by a second, smaller and more 
limited transfer of training study usihg a second 
' group of pilots with varying degrees o^ experience on 
the particular task, who^ will be given ^simulator 
training before* performing in the aircraft. ^ The per- 

foxmance-JneasurP!^ frojii— the— first study, would be 

related mathematically to the transfer bccurririg in . 
the second study ., The. intent is to find. an equation 
that will enable us to predict and , safely extrapolate 
from the data least expensive to collectl 

b. The other in which a transfer of training study is 

..conducted (without a preliminary per formance experiment) ' 
using economical multif actot data collection plans 

for the simulator training phase with equal or fewer 

conditions . tested later in the f ligh^t .pha^e. Economy 
is effected- through the use of sequential 'data' coll^.c- 
tion strategies and the. reduction of inraircraft tests. . 

In both approaches, the data coLlection effort \is reduced . 
(and economy is effected) as the costs in time,' moneW ^ and 
.difficulty of each phase, — performance ^ training, or flight 
increases , A fundament:*!- assumption in the proposed^ approaches 
is that a poor model of a. relatively complete .multi fc^ctpr study 
will give iuoie accurate —arrd— t ^selul ♦^ • jieclicLion daLa — than a 
better model of a severely limited part o.f the overaia space. 



NAVTRAEQUIPCEN 7 7-C-0 065--1 



ERIC 



A number of specific plans associated with each approach 
pe conceptuMi-2ejd..aS--aho:wii-Jj:i.Jr-a);a-e-^^^ 
m cost and probably, effectiveness. Which one would be used 
depends upon the circumstances at the time, i.e., the available 
resources (men and equipment), time,, and above all the dedication 
of those involved to. the research effort. 

PERFORMANCE TO TRANSFER APPROACH (.1) 

These approaches all use the results of a, complete multifac- 
„ tor performance study to select or otherwise minimize the number 
•of conditions that need be included in a transfer of trainina 
study. ' • • ^ 

Selected Configurations (Plan I-A ) * 

A complete multi factor p erformance experiment would bb 
performed first in the simulator.' Pilots would be' used who were 
sufficiently skilled to minimize an extensive training period 
in order to fly__ the simulator. They would, however-,- -fatl- into 
at least two groups with' high and low^ experience in making 
carrier landings, (or what&ver the experimental task may be) 
Two or more levels of task drfficul.tr WuK also be included 
Multif.actor^systematic screening designs in the paradigm proposed 
by Simon (1977) would be used for this study to make the data 
collection as economical as posbible. Multiple performance • 
measures (i..e., dependent variables) relevant to the task which 
could also be measured in the aircraft would be used. Additional 
measures might also be taken. 

a classic trans fer of t r ainin g experiinent would-be-^ 

performe-d-m^ep^Sdently of the performance study., New pilots 
would-be selected, with minimum carrier landing 'experience 
but with one group being high skill pilots and another belAg low 
skill; pilots. They ..would , all be trained first in the simulator 
and later tested in the aircraft in flight. • • 

The particular configurations to be used in the transfer ' 
■experiment would be based on a study of the results of the 
performance experiment. For Plan 1-A no other use of the 
performance data jas" it relates to the transfer study) ,i-s planned 
The .purpose of this approach is to limit the number of configu- 
rations- to be used in the transfer of hradning study to only the ' 
niost;.interesting. The ex;act number depends., on the resources, 
the -information desired, and. any formal requirements of the 
"experimental designl. 

. Criteria for selecting' particular'' configurations micrht ' 
include : ■■ > - - , ^. ; 

•a. Pe ^lQ£mA ncg^_gj;.p J ,_AChi ^4^ ..^^-ei^g-tt^ nn wli'lnh 

high, low, and medium performance' -levels were , -achieved ' 
might be selected to see^ to what extent transfer effec- 
tiveness correlates' with performance effectiveness. 

4 



TABLE 8, APPROACHES TO ECONOMICAL TRANSFER 
OP training' RESEARCH 



I. RELATING PERFORMANCE TO TRANSFER 



Performance Studies 



Transfer of Traininc Studies 



(Simulator Tng. 



- 4 



Aircraft Flight) 



PILOT 
TYPES 



, High Skill Pilots 

a) Minimum carrier landing 
experience ( 



b) Maximum. carrier landing 
experience 



Minimum Carrier Landing Ex perience Pilots 
a) High skill pilots 



b) Low skill pilots 



APPROACH 



Co]ijctJh^MaJ:a_requir.ed. 
to develop a full model 
multifactor simulation ' 
performance map. 



(continued) 



JLAN-i-A r -SeleGt-a-f ew-si-muiator-configurations--to— 
investigate on the basis of an exaraina- 
' . tion of the performance map. ■ Include 
additional cot! figurations of practical 
and scientific interest related to simu- 
lator design problems. With these dp a ■ 
classic, transfer study. 

PLAN 1-B. Systematically develop a transfer response 
' surface using a fractional factorial 

design of low resolution. Use it to pro- ■ 

vide criteria for writing a transfer- . 

prediction equation from performance data. 

Validate.,' 



TABLE 8.. APPRO-ACHES/ TO ECONOMICAI* TRANSFER 
OF TRAINING RESEARCH (cont.) 



II. PERFORM LIMITED DIRECT TRANSFER OF TRAINING 'STUDIKS ■ . ' " ' i 

I ' „ , Transfer of Training Studies 

; '■ \ ' ' ■ . ' 

; . ■ ■ ■ ■ \ ' ' ♦ ■ 

PILOT . i , ■■ , Use low Skill pilots wil;h no carrier landing 

TYPES ,; ^ . expe'rience (or whatever population the transfer ' 

j data is to be generalized to] : - all receive . ■ S 

i ■ . . ' ' simulator training plus flight time. 

■■ ■, ■ PLAN II-A. Use "new Iparadigm" for economical'' data ■ 

|. ' ' , , collection to empirically map a trans- 

i /■ fer surf ace,. building sequentially 

y. '/ ■ each data point involving both training 

APPROACH ' I . , / ' and flight time; until an adequate 



■ - modB-l~i"S-TepresentedT 



0^ ■ . I / . , ■ ' ,, to search for configuration yielding 



PLAN II-B. Use evolutionary operational approach 
to search for con! 
maximum transfer. 



PLAN II-C. Develop full model, raultifactor training 
map, but limit continuations to flight 
to a minimum fractional, factorial design. 
„' ■ ' .Use the flight data to provide criteria ' 

■ , ■ ' for writing a transfer-prediction 

^ I . ■', equation frpm training data. Validate. 
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Engi neering cost advant ages. How much do simulator 
configurations on which peirformance levels are prac- 
tically the- same but which differ considerably i;a 
production costs differ in transfer effectiveness? 
Is an increase in transfer actually cost effective 
from 'an engineering point of view? 

Engineering' state-of- the-ar t advantages . Some config^ 
urations produce reasonably adequate simulation and 
acceptable in-simulator performance levels without 
straining, the state-of-the-art. Other configurations 
may requii^e additional engi-neerina develooment to 
advance the statfe-of-the-art but- may be less reliable 
and more costly to operate or maintain. How do they 
differ in regard to transfer effectiveness? 

Correspondence with r eality. To what extent does 
"fidelity" of simulation affect transfer effectiveness? 
If we select configurations that approximate. re.-.lity> '■ 
2J-1 .and-nrpt-well-, ° is tr^ansfer effectiveness TnaTrkedly — 



dioffetent in the two cases? 

e- "Scientific" knowled,(^e . The investigator might 

include any configurations. that might increase his ' 
. ^ understanding of . the transfer process , -particul^^rly as 
■to how It relates to the performance effectiveness. ... 

It. is not. possible, to list all the; detailed questions that , might 
.be. investigated. They . will have to be. determined by the pattern 
Of the performance response surface,, the imagination and' 
curiosity of the investigator , as • well as his knowledge of the 
problem. Furthermore J such .decisions will be limited hy. the 
time and money available' for the follow-up transfer study. 

The fiv4 criteria' listed above are. probably not completely 
orthogonal. Tor example, the, most realistic . configurat^ion might 
be the most costly, the most complex , and the most unreliable. 
Still, they are representative of^ things an investigator may 
wish to explore for the transfer problem. * . . • ■ 




will be examined As many conf igurations , as the investigator 
wishes would be examined in a./simple one-way AN0VA design; each 
configuration is treated as a qualitatively different condition-. 
The disadvantages of this pl^n are U. it. reqiaireis redundant in- 
formation to be Collected; each condition must be repeated a 
sufficient number of times to provide some reliability to the 
means, and 2) th'2 manner In -v^hich configurations are selected - 
increases the chance that important oonf igurations -will b^ ' 



overlooked and important relationships missed. -This approach, 
at best, is a make-shift :One , and certainly.' the -one least likelv 
to b6 effective. It is expedient, but where long range planning, 
is possible, some other appi^oach. should^' be" employed .' ' 



. . ' NAVTRAEQUIPCEN 7 7-0-0 065-1 

Performance-Transfer Prediction Ap£yro ach - (Plan I-B ) 

The purpose of this approach is to develop an equation that 
would .predict transfer effectiveness from simulator perfoirraance 
measurements. If a valid prediction equation could be ^established 
~febe--pe-r~fo-rffia-n4s^a-data, which is less expensive to collect / would 
be used to estimate the transfer effectiveness of configurations 
not^actually studied in the experiment and possibly of other ' 
simulator configurations involving similar tasks, 

*••.•' . ■ • \ 

With a group of skilled pilots^ a multif actor simulator 
performance study would be performed. Since excessive . simulator . 
training would'* not be required, this phase of the plan should.be 
as c-.mp-lete as possible,. This per form^nce experiment is 
idencical in process and result with that obtained in Plan 1-A. 

The tran 5fer, phase of the plan would differ from Plan 1-A, 
Training and, flight tests would be conducted by a different 
pilot on each configuration, but the 'configurations^ would be 
selected in a sy3teruu\;:ic manner to * take advantage of the economy 
offered by :.hternal replication rather than redundant replication 
of the sams conditions. The intent would be to employ a minimum 
fractional lactorial plan to create a transfer map over ,^the same 
experimental space that had been covered by the performance, map, 
.However, the transfer map would\be represented by a lojwer-order 
equation, and might only roughly\ approximate the true'^ transfer ^ 
response sur/ac^. . .. * 

Transfer data would be ' collected in a series of small blocks 
i*e,, differenu "small fractions of a total factorial. As each 
block is collected, the sum total of data up to that' point would 
be. correlated with the complete data- from the performance maps 
to see how. strong. a relationship could be- found. Presumably as 
the transfer response surface ,is more completely approximated, 
the more likely the relationship betweeli it and the perfbrmance. . 
response surface can b^ used J^or prediction purposes. However, -, 
the intent is to stop before too much transfer data 'has 'been 
collected when additional improvement seems unjustified for pre- 
diction "purposes , The assumption is made that even a poor 
approximation of a rather complete multif actor transfer surface 
will ultimately^ enable a better predictiofi — ^^ 'operationally — 
from performance data than were a limited transfer surface 
approximated, 

* 

At least two methods of relating the performance and 
transfer data might be tried: 1) to correlate only the responses 
from corresponding configurations in both sets of data; 2) to 
first use the collected transfer data to estimate transfer 
values at configurations at which no empirical data had been 
collected but which correspond to configurations used in the 
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performance/ S|l:udy ; then corrola to os timated and empiricai ..transfer 
data with perjf ormance data . Otho r reasonable " varia^ioTis^ on these 
techniques could be tried- The proportion of variance by which 
the tw.o sources of data overlap serves as an indication' of the 
strength cf their relationship- 



Several tonditions might operate to make the relationship 
between thee two sources low. One, the model of the .e4uation . f rom " 
the transfer data .may not — because' of the small amdunt of ^ata . 
allotted to that segment of ^ the investigation be.^omplex 
enough order ^to make accurate eiLimates, The sequeritial approach, 
however, would allow the model /to be built a block at a time 
unbil it is/optimized if the time, and money available permits 
it. Two, thepre may be enough data, but in the wrong metric scale; 
data transformations would be ^-equired. Three, ot^er factors 
than simulator, performance may/ Eect the level of performance 
in the aircraft and the trans^'ef. effectiveness measure. This 
means that bnpe no further in^ireinenl:- in the--rel-abionship^^ 
achieved by ci^hartcing the mod^fe^l of the transfer data, the in- 
vestigator wijll want to look /'for other factors such as simulator ' 
fidelity and task di f f icul ty / that might account /for unexplained - 
sources ,of variance. An important part of this /study would be 
the validation of the derive/d question. The transfer effective- 
ness of other 
tion checked 



configurations 
empirically • 



would be predicted/ and the predic- 



Another factor that mi(|ht account for the/low relationship, 
if one is found, is the dif jferen:;e in the pilot populations 
that were useld to get the ptfer f on lance and the /transfer data . 
.Ordinar<iiy mojre skillful pilots may be used in the simulator 
■performance sjtudy when minimum training is inyolved . than in the 
transfer study where extensive training may b;3 needed. Problems 
of interpretation might arisAi i£ c:onf Iguratidn-by-pilot- skxll/ 
experience inteiract Ions were \to occur but could not be isolated. 
Therefore, unless all pilot combiiiationr; are! to be included, 
pilots from tfjie same populatiqn.y should be u;sed for the perfor- 
mance and transfer phases in this approach. / 
^ ■ ' i • \ ' / 

Since we have had no experVence calculating these relation- 
ships, we musq be prepared for them to be low. While it seems 
reasonable to bxpect some kind o\f relationship to- exist, even 
wit.h other intervening, covariant factors, ;it may not be 
sufficient for i prediction purposes. If it, turns out that no 
relationship cAn be established, that itself would be an impo;r- 
tant finding. 



LIMTTMI) DIRPX'T [rRANSFhlR APPROACFI 

1 , 

In tills ap^);oach, no initial pbrformfince shudy would be 
•perf orincH.l . Instead, we wouJ.d star t \ i nmiqdia tely with a multif actor 
triinfifcr of t rain iiuj oxper Lmenl: uslii\q tho strategy for economical 
data cy Lice tion described earlier for the construction of a 
per Co finance map.' \ 
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ComiHete Transfer Surface ( Plan J 1 - A ) 



For a given pilot' population and task, each pilot would 
betrained on a particular simulator configuration, after which 
his performance in the air, would be tested. The transfer 
effectiveness of each pilot/conf icfuration combination would be 
calculated separately.. The simulator configurations, represent- • 
•ing experimental conditions, would be selected and used according 
to the. "new parddigm" described for economical multifactor 
research by Simon jl977b). To keep the study as inexpensive as 
possible, the principles of sequential data collection would be ' 
.employed, starting with minimuni-oi:dor designs and progressing 
until the model. adequately fits the responses. , However, the intent 
Of this plan is tfo create a complete transfer surface.,' 

^ . Initially simulator configurations would be selected to 
provide a Resolution III design. Theoretically, we can study 
the effec ts o f N simu lator factors with N + 1 pilot/conf igura- 

""Eions _af there are no interaction effects among factors. Since 
two. factor \interactions are common in behavioral research, the 
-investigator will probably continue the data collection ox\ ' 
(N + 1) new pilot/configurations in order to isolate main from 
two factor interaction effects. Of course, inspection of the 
first block of data may negate or modify the second step. After 
an>J.nspectipn ofthe new data (combined with that from the first 

.block) , the investigator may wish to add other configurations to 
determine a second order response surfac¥ (if. the factors are 
quantitative and continuous and if that accurate a represen- 
tation IS justified) . The investigator always has the option 
of continuing or stopping. 

The advantages of thi& approach are that it is direct, 
-relatively uncoinpliCfited , and. the most economical way qf collec- 
ting data for the amount of information indicated. Since each 
data point is collected independently of^"the others — a dif- 
ferent pilot/configuration being 'used on each — scheduling and 
other logistic problems are simplified. The immediate informa- 
tion obtained is a measure of transfer effectiveness and the 
response surface is a transfer surface., .. ^ 

. The disadvantage of this approach is that although the 
paradigm is- the least costly data collection plan for the amount 
of information obtained, being a transfer of training study, it 
IS still ej^penslve. This approach does not offer the opportunity 
to develop equations thac' might permit predictions to be made ^ 
from prior performance . studies , which can usually be done far 
more economically than a transfer study*, byit possibly not as 
accurately. _ > ^' 
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Search- For- Optimum Transfer (II-B) 

If many 6f the critical simulator factors were quantitative 
and i t were ppssible to examine configurations at continuous , 
points between the extreme ranges of interest, then a search: 
strategy employed in industry to optimize production yields 
(EVOP) might be Used to search for the most transfer-effective 
configuration • While many factors may not be quantitative or 
continuous^, .there are usually sub-groups that can meet this' 
qualification which mi ght be investigated in a separate study 
once a more gross, overall transfer pattern has been developed.. 

■ \- ■ 

The basic experimental design begins much like a screening 

design^ being a -^ solution IIT 2^"!^ design. The main difference 
is in. the range that is covered by each parameter of the design. 
In screening designs, one tries to encompass the total effec- 
tive operational range of the experimental variables immediately. 
The assumption is made that this can be estimated and that the 
relationships within" those limits can ordinarily be approximated 
by a second degree polynomial. In search designs , the investi- 
gator starts by looking at only a small part of the total space 
of interest. He tries to guess where an optimum might be, but 
he does not 'attempt to cover the total range. Instead, he looks 
at a part of the total space and uses that data to estimate where 
to look next, each time approaching closer to where the optimum 
configuration for maximum transfer would lie. This continues 
until he locates it. The method would be used when the surface 
is too complex to be cover*ed by a single design and the investi- 
gator has little idea of where, the optimum might be. 

A transfer of training study — simulator training and 
aircraft test — would be performed at the minimum number of 
conditions (i,e., simulator configurations) required to include 
all factors in a'Resolution III fractional factorial design. 
Either the Box and Hunter or Plackett and Burman plans might be 
used, the latter in some caser> requiring fewer data points. 
The space encompassed by the experimental points would be only 
a small part o^ the total space of operational interest. A 
different pilot. would be tested on each condition. The results 
of this initial data collection effort, in the fofrm of a first 
order polynomial, would be used to estimate the direction, away 
from 'the space covered^ by the original study, in which the con- 
figuration yielding tlr{6 greatest amount of transfer is likely to 
be found. (This, of -.-course assumes that it is not ^within the' 
'space originally examined.) A second set of observations (Reso- 
lution III) would be made at new coordinates in that vicinity^ 
This procedure would be repeated until the observations appear 
to surround the location of maximum transfer. 

A difiadvantaqe of this plan, when it can be used^ is that 
it seeks a point of:. optimum transfer. Seldom in human factors 
work is a single point sufficient information, since design 
decisions must ofl:on be compromises among performance, costs. 
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and other practical considerations. At times, optimum results 
may take the form of a ridge of equal performance, in which case, 
some trade-offs could be made. Were the effort worth it, a 
response surface- might be completed for. the space around the 
optimum point. Additional, data would have td'Te taken to fit 
the surface to the model correctly representing the complexity 
of the surface. - . ' 

Reduced-Eli ght Predicted Transger ( II-C ) . ' 

In this approach, economy is achieved by reducing the amount 
of flight data that would be required. This would be accomp- 
lished in one of two ways: 1) to predict transfer effectiveness 
of simulator configurations that were never flight- tested by ■ 
u6iog equations representing the response surfaces that were 
derived from transfer data (based on training and flight test) 
made '*on only a few configurations; 2) to predid't transifer 
effectiveness from performance data collected during the tradLnxng 
period after the relationship between training performance and 
flight performance has been established. These two approaches 
employ features that are similar to the Search Approach and to 
the Performance--Trc ^fer Prediction Approach,, respectively. 

In both cases, complete transfer studies would be performed 
on the configurations making up a Resolution III design, 'if 
time and money limitations permit, a higher resolution design 
would be. employed involving more experimental conditions. 
Training perforniance data would be, obtained, followed by the 

^ flight test data. Transfer effectiveness values, could be calcu- 
lated for all of these configurations an'd a first order, linear 
polynomial could be written from the transfer- data tha-t could be 

'^^3^ predict transfer effectiveness for other configurations. 
How accurate this prediction would be depends on how well the 
equation approximates the response surface. If one itmst extrap- 
olate beyond the boundaries of the original experiment^ pre- 
dictions could be quite inaccurate. ■ \ " , 

with that data from this limited study, however, the 
investigator would have a second means of estimating transfer 
effectiveness. could take performance measures collected at 

, different stages oV the training phase and" see how they correlate 
with gerformance • in the aircraft (or transfer effectiveness). 
This correlation, as an equation, could also be used to predict 
transfer effectiveness for other configurations provided the 
training data were made available on those configurations . 

, Of course, these descriptipns o,f both techniques are over- 
s'lmplified. It is unlikely that high correlations will occur 
without additional Work on the part of the investigator , Quite 
probably othgr' parameters , e.g.^ f ideli ty , task ^difficulty , pilot 
q^kill/experifence, wquld have to be introduced as multiple preA . 
dictors to improve the estimates of 'transfer effectiveness. \ 
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Perhaps^ the two measures combined into a single equation m4.ght 
provide a more accurate prediction. It may be that the* 
prediction is only suitable for 'ranking a set of configurations " 
but .not. for measuring the actual amount of transfer. "fhese are 
all experimental questions that can only be answered empirically 
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SOME UNFINISHED BUSINESS,'- MEASURI^MENTS AND GRITERIA 

Certain ques'tio'ns associated VN^ith performance and transfer 
measures remain unanswered although the answers to each affect/ 
to some extent, the usefulness of the proposed methodologies as 
well as the very effective'ness of' the AWAVS human performance ■ 

research program, . • , ' 

•• • ' • ^ , 

■ . First, there is, the question o'£ what performance measuires 
will be taken on both the ptiy^ical. system and this p?-lot system? 
The usefulness of the experimenta^l res^ilt's 'depends on how- 
relevant the measurements made in the experiment a;re to the 
operaiiional task. More numbers , taken becausfe tjt;iey are more -. ■ 
expedient or convenient, do not: guarantee that the results ofj. 
the study will be useful or- eve*n correcr. insofar as the opera- 
tional situation is concerned. Will performance datasbe" 

--av^i-taixle- to the rnve-s-ttgcTtDT duTifrg "li^'^run 

following the run, by the time a second pilot cis to be run, » , 
or whdn? Will there be the 'capability . of performing summary 
analyses on the raw data? How quickly might that be available? 
Advanced experimental methods are economical because of their 
sequential nature. That means that they rely or; a process 
whereby a small block of data is collected and examined 
(analyzed-^ to dete'rm^-ne' if and what' subsequent steps are needed. 

■ If this process is delayed beyond the'^time it takes to set up . 
for the next trial, the data collection period is, not only 

_drawn_ out . inef f rciently but^the ef f ects of--- the delay- on the 
pilot couTd conceivably distor;t his perf ormance ,v , 

Another problem related to measi^rement in a transfer of 
\ training study involves the critor. icn of training employed. 
Will t .e interpretation of ti->e res'jits differ if we use time-to- 
criterion, or 'if we use. equal number of training . trials , or if 
the criteria we employ (as we should) are multiple response 
measures? Associated with these questions are others, such. as: 
how.does the use of different criteria affect the reliability 
of the results, the logistic problems of .running the experiment, 

and so forth? 

\. ^ . • _ . ■ ■ - 

'*A third problem relate'd to measurement has to do with the 
preferred form of meaF" ""^ment to be employed in the analysis. 
While we are ultimately interested in transfer effectiveness,' 
data expressed in those terms" are in fact particular transfor- 
mations of performance scores! It is necessary to discover 
whether pretiictions might be more .easily and accurately made if 
more basic performance measures were employed, leaving particular 
transformations up to the userti ol: the data. For example, we . 
may find th^it performance in the aircraft can be predicted from 
performance in the simulator more readily than transfer measure- 
»*nents in the ai^-craft. Then again, we may not. - 
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/.Measurement problems, are fundamental to any research , 
conducted on transfer of training and to ignore them or assume 
that. previous research has resolved these ^questions can onl'y 
increase the risk that our experimental efforts will fail. 
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GLOSSARY 



Screening and other fractional 

factorial designs (see below) • =' 
do not isolate all main and / 
interaction effects from one ■ 
another. ' A comparison which . 
intends to isolate one effect 
may therefore also include, 
estimates of others. When two 
or more effects -are 100 percejit 
con f dun d'e d in ~th i s way /• t he"7 
■ effects are said to be aliased. ' 
The estimated effect is actually 
the sum effect of the aliases. * 
(See. also, FRACTIONAL FACTORIAL 

^ DESIGN; CONFOUNDING). 

(Sometimes referred to as carry- 
over, cross -over, or residual . 
designs). These experimental 

. designs are used when a subject 
is tested sequentially over a. • ^. 
number of experimental condi- 
tions. These desigr^s are -con- 
structed so as to isolate the 
direct effect of a treatment 
from any residual effect that 
may, have been "carried' over" 
from the previous treatment. 
Change over designs are dis- . 
tinguished from serially balanced 
sequence designs in that the neces 
sary jDalance required to i/solate 
direct and residual effects is 
distributed among a number of' 
subjects in the chaage-oyer 
design but is complete within a 

. single subject for the serially 
balanced sequence des'igh . (See 
also, SERIALLY BALANCED SEQUENCE 
DESIGN) . 

When estimates of the effects of 
two or more sources of per for- ^ 
mance variance cannot.be com- 
pletely ^isolated, either inten- 
tionally or through faulty ex- 
perimental 'QO^ign, the effects 
asre said to be confounded. Con- 
fouTiding may range frrom some ^ 
minimal percent up to.. ICQ percent, 
(i:ee also ALIAS) . , 



75 




NAVI^RAEQUIPCEN .77-C-0065-1 



.'V 



FACTOR LEVEL CHANGE NUMBER 



When experimen"hal conditions are ' 
run se^^uentially , the level or 
setting of each factor rpust be 
changed from time to time. In 
screeninr designs,., the "change^ - 
number*! indicates the total num-^ ^ 
^ .ber of; times a particular 

factor ' must be switched betv\heen 
: its -high and low levels. It is 

important in the. design of an 
; : experiment when making the change 
.is difficult or'otherwise costly. 
(See also SCREENING DESIGN), v 



FRACTIONAL FACTORIAL DESIGN 



HALF-N0PJ4AL PLOTS 
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This experimental design is com- 
/ posed of ^ some fractional subset 
of the total number of experi- 
mental conditions in the com- 
■pliete factorial. It is employed 
when certain effects (gehera'lly 
higher-order interactioris) are 
i expected to be negligible or non- \ 
! existent. Subsets of experimen- 
; ;tal bonditions 'for the fraction 
are selected -in a way that allows 
the comparison- "for the negligible^ 
effects to be us^d to measure the 
ef^S^ts of additional factors 
aliased with them.' Fractional 
factorials of two levels are com- 
monly designated in the form 
2^"'P. For example, a 28""4 • ^ 

fractional factorial wo.uld be a 
2 or a 1/16 fraction of a com- 
plete factorial. That is, a 
• particular subset of 16 condi-,^ 
tions out, /of a total .of 256 would 
be used to 3tuldy . eight factors, 
/at two levels each.j; A "satu-- 
. rated" fraction"^! factorial ' 
design' is one in which there are 
n observations for n-l main 
effects 

This graphic technique is use^—to 
identify ' visually the critical 
Effects of 2 factorial orv2^"P 
'fractional factorial experiments" 
that have beei;i plotted in-drder 
of absolute magnitude on half- . 
-normal plotting paper. (See ^ ' 
ialso FRACTIONAL FACTORIAL DESIGN) . 
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HOLISTIC 



,1 



multifActok^ experiment 



A philosophic point of vij6w m 

the conduct of behavioral experi- 
ments that emphasizes /the impor- 
tance 'Of; accounting" fot a's irianY;: 
. critical variables as t^ossibley^* 
whethfeir^equipiTient eriVironment , 
subject"^ or *tempora/Lv controllfed^ 
or uncontrolled. ■ ./mplementing . 
such a philosophy 'requires the , 
.'applicatiorf- of » principles of 
economical multif ax^tcn: -designs. 
(See also REDUCTIONISTIG)'. 

As lise'd in this report, a multi- 
factor lexperiment is one which 
attempts -to satisfy the holis- 
tic ph-ilosophy. Thus, a thV.ee 
. or even! five . f accor ''experiment 
- (at the i'fcjeginning; of a research 
ptogrMttij'v while involving mul- 
tiple '/factors, ,.''wpuld ncjt or'di- . 
narily' tS)e a miiltif "actor experi- 
ment as jthe te-rm is used berj^. 
CoTnpromi|ses with non7experimen- 
tal cond^itions surrounding an, . 
. 'experiment may make it impossi- 
ble to liinqludfe- aH potentially ^ 
critical^ /iacTtors ? but" th^ inj?'- 
tiai '■-eiyphasis' will be on trying 
to do SCI (See also HOLISTIC), . 



ORTHOGO;>IALITY 



PERFORMANCE . EXPERIMENT 



That property of an experimental^ 
design 'v^h'i.ch insuresi that the 
different effects shall be capa- 
ble of direct and separate, 
estimation without any ccn^ound- 
. ihg. Thte suilns of squares of ' 
all ef fefets will be independent 
and additive. jSee. also. CON- '.^^ 
FOUNDING) . ^ . ' 



^As the term i^s us^ec 
"""^^feh-j^^ ^ f P rmanc e^xi 



id in this report 
4^^^,wj.Aiic*ii^^^xperimeYit ,^ is one 
that ijieasures operator/system 
performiance under .orie set of 
condx/tions, presumably uninflu- 
enc^jed by amy^ o1;her .prior condi- 
tions . Measuring pilot perfbr- 
ma ne'e* in a simulator jijrith di!£-.. 
ference configurations could be • 
an Example of this type, as « 
opFjbsed to another type referred 
to /as a ''transfer" experime/it. 
(See alscTRANSFER gXPERIMfiNT? 
QUASI-'TRANSFER ^'EXPERIMENT)'. * 
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PRINCIPLE OF PROPORTIONAL 
' PpQUENCIES 



QUASI-TRANSFER EXPERI?4ENT 



A necessary and sufficient condi- 
tion that the main effects of 
two factors be uncorrelated is 
that the levels of one factor 
occur with each of the levelsv 
of the other factor with pro- ; 
pprtional (not necessarily ; ' 
i equal) frequency. * ^' ' ' 

This is a transfer* experiment in 
which performance is never . , 
^ ^measured onder realistic, i.e. , 

' non-simulation , ''"'conditions*.. • s 

• For pilot traininc, this meansv 
that the experiment would in- • 

: elude no post-training *period^ 
in v/hich performance was mea- 
sured in the aircraft. Instead 
an alternate simu]_aticn configu- 

' ration would be employed tOx 
'represent flight conditions. \ 

■ (See : also, T.^ANSFER EXPERIMENT ;\ 
PERFORt'lAN^E EXPERIMENT) . 



REDUCTIONISTIG 



A philosophic point of view in • 
the 'conduct of behavioral ex-^ 
periments that advocates^reduc- 
ing the variables in' an , experi- 
ment to the. smallest number 
possible. In its extreme form 
the resulting experiment is 
one which a single factor, 
is varied and all^other sources 
of variance are field . constant. , 
This philosophy is in direct^* 
opposition to the holistic 
philosophy. (See., also HOLI^STIC) 



RESOLUTION 



78 



design of "resolution" ,R is one 
in .which no p-'f actor ef fe.ct is 
confounded with any other ef-- 
feet containing *f ewer than R-p 
factors. The resolution of a 
design is noted by the appro- 
priate Roman numeral as a sub- 
script in the fractional fac- 
torial"' designation, e.g. / 
28^4 design. 71 design of 

Resolution III does not con- 
found main effects with one 
another^ -but does confound 
therta with two-factor interac- 
tions. A design of .Resolution IV 
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RESOLUTION (Continued) 



RESPONSE ^5URFACE METHODOLOGY 



SCKEENING DESIGN 



isolates main effects from one 
another and from two-factor 
interactions, but 'the two^ ~[ 
factor interactions are aliased 
in strings, A design of Reso- 
lution V isolates all main ef- 
fects and all two-factor inter- 
. actions from, one another. In 
all screening designs, main 
. effects and two-factor inter- 
actions are confounded v/ith 
higher order eff^icts. (See , 
Ulro, FRACTIONAL FACTORIAL 
DESIGN; SCREENING DESIGN) . 

This refers to a particular 
strategy introduced and pro- 
mo ted ^by G,E.P. Box and . asso- 
ciates for conducting experi- 
ments to obtain an equation 
_ representin'g the response multi- 
function, or surface. --It is 
not a design, per se , but the 
judicious use of principles of 
blocking, fractional factorials 
and tests of model adequacy in \ 
a way that insures an accurate 
representation of performance 
within the experimental space 
at minimal data collection ccst_ 

As used in this • report it refers 
to a saturated or nearly sat- 
urated fractional factorial 
design capable of handling a 
j large number of- factors. These 
.designs are all of the foriti/ 
2^"*?^ generally of Resolution 
III or IV, . The initial infor- 
mation is first evaluated before 
subsequent data are collected, 
ths^purpose being only *to iden- 
tify~the~-critic'al factors within 
a larger candidate group. Addi- 
tional ^ata must be collected' 
ordinarily to meet a , second and. 
separate purpose, defining the 
response surface. (See also^ " 
FRACTIONAL FACTORIAL DESIGN; . 
RESOLUTION) . 
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SERIALLY .8ALAI>JCED SEQUENCE 
DESIGN 



modified change-over design 
for isolating direct and 
residual effects/ in which the 
necessary balance occurs with- 
in the extended nuir.ber of 
t^riais run by a single subject. 
This! contrast^ with the change- 
over design in which the bal- 
ance is .obtained among several 
subjects each tested on fewer 
trials- (See also CHAIIGE-OVER 
desIgns) • . \ 



/SINGLE FACTOR EXPERIMENT 



/ 



This r.efers to the typ 
periment proposed by 
Reductionist • hk us 
report, it need riot 
factor, but for a!ny 
number which is a se 
incomplete number' of 
tentially critical f 
affecting the partic 
formance, (See also 
JSTIC) - '\ 
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REDUCTION- 



TRANSFER EXPERIMENT 



TREND-ROBUST EFFECTS 



In contract with a performance 
experiment, as used here, this 

. refers to experiments in which 
interest centers on- the resi- 
dual effects -that practice on 
one set of conditiohs has ' , 
on the performance of a. second : 
set which follows^ |-(See also,'. 
PERFORMANCE EXPERIMENT; QUASI- 
TRANSFER EXPERIMENTi . ( 



Designs exist that iscilate linear, 
quadratic, and/or cubic trend 
effects from experifr|ental^, :6f- 
fects of -intere^j.t. Examples of . 
trend effec are subject 
learning, or equipment drift 
over time* A trend-robust ef- 
fect is "one which isj not bi- 
ased, or only minimally biassed, 
by* trend,s running, through the 
data- 
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